Multilevel Modeling of the Effectiveness of Deep Learning-Based Adaptive Learning Systems

Abstract

In the context of the digital transformation of lifelong education, adaptive learning systems (ALS) based on deep learning algorithms are becoming a key tool for personalizing educational trajectories. Despite the significant growth in related publications, the issue of the multilevel effectiveness of such systems under rigorously controlled experimental designs remains insufficiently explored. The objective of this study is to develop and empirically validate a multilevel model for evaluating the effectiveness of deep neural network- based adaptive learning systems using a randomized controlled trial (RCT) design. A total of 842 university students participated in the study and were randomly assigned to an experimental group (n = 421) and a control group (n = 421). The experimental group used an adaptive learning system built on a hybrid Transformer-LSTM architecture with an attention mechanism, whereas the control group studied through a traditional digital learning platform without adaptive algorithms. Hierarchical Linear Modeling (HLM) was employed to assess individual, group-level, and institutional effects. The results demonstrated a statistically significant improvement in academic performance (β = 0.47, p < 0.001), cognitive persistence (β = 0.39, p < 0.01), and metacognitive regulation (β = 0.42, p < 0.01) in the experimental group. The effect size (Cohen’s d = 0.68) indicates moderate-to-high practical significance. Multilevel analysis revealed that 27.0% of the variance in learning outcomes was attributable to between-group differences, highlighting the necessity of multilevel statistical modeling. The findings support the hypothesis regarding the high effectiveness of deep learning-based adaptive systems and provide a methodological foundation for their large -scale implementation in higher education institutions.