Emotion Recognition in Online Learning Using Multimodal Artificial Intelligence

Abstract

In the context of the rapid advancement of digital educational technologies, the personalization of online learning based on learners’ emotional states has become a critical research challenge. This study proposes a multimodal emotion recognition system based on the integration of visual, auditory, and textual data using deep learning techniques. The objective of this research is to develop and evaluate the effectiveness of a multimodal architecture that combines Convolutional Neural Networks (CNN), Recurrent Neural Networks (LSTM), and transformer-based attention mechanisms to improve emotion recognition accuracy in educational environments. The experimental evaluation is conducted using widely recognized datasets, including FER2013, AffectNet, and RAVDESS. The proposed model demonstrates strong performance, achieving an accuracy of 0.89 and an F1-score of 0.86, significantly outperforming unimodal approaches. The results indicate that the integration of multiple modalities produces a synergistic effect, enhancing system robustness against noise and incomplete data. Additional experiments confirm high discriminative capability (AUC > 0.9) and model stability under conditions of missing modalities. The practical significance of this study lies in the potential application of the proposed system within adaptive educational platforms, enabling dynamic content adaptation based on learners’ emotional states and thereby improving learning effectiveness. Overall, the findings confirm that multimodal artificial intelligence represents a promising approach for the development of intelligent online learning systems and adaptive pedagogy. The proposed architecture demonstrates high accuracy, robustness, and practical applicability, highlighting the effectiveness of multimodal approaches in emotion recognition within digital learning environments.