Abstract: The current solar flare forecasting models are mainly based on the statistical relations and the use of the sunspot group feature parameters extracted from the magnetogram as inputs. Thus, the level of the system autonomy is impaired and the precision of the model is lowered, since the high order elements and the more abstract features contained in the magnetogram are missing there. This study, by combining the advanced artificial intelligence technologies and the massive sunspot-related data, such as the magnetograms, the corresponding feature parameters, and the tags of the flare events, establishes a multi-modal solar flare forecasting model based on the deep learning techniques. Specifically, the model takes the advantage of the high precision of the fully connected neural networks and the high recallability of the convolutional neural networks which can effectively extract the high-level semantic information. Experiments show that the main evaluation indexes of this model are at least 7.8% higher than those of the other models.