Automated System for Objective Assessment of Educational Achievements Using Generative Artificial Intelligence

Abstract

In the context of the systemic digitalization of education, Automated Scoring Systems (ASS) based on Generative Artificial Intelligence (Generative AI) are becoming an important instrument for scalable and objective assessment of educational achievements. Despite the high efficiency and operational flexibility of such systems, the fundamental psychometric issues of their reliability and validity remain insufficiently explored. Of particular importance is the transition from Classical Test Theory (CTT) to mo dels of modern measurement theory, specifically Item Response Theory (IRT), and particularly the Rasch model, which ensures parameter invariance and metric comparability of measurements. The purpose of this study is to conduct a comprehensive evaluation of the reliability and validity of an automated scoring system based on a generative language model using the Rasch model as an analytical framework. The study applies the one - parameter logistic Rasch model to analyze item characteristics, the distribution of examinee abilities, fit statistics, separation indices, and the detection of Differential Item Functioning (DIF). The methodology involves simulation of a dataset consisting of 1,200 students and 40 open-ended items automatically evaluated by a generative model. The analysis is performed using maximum likelihood estimation procedures and the calculation of Infit and Outfit statistics. In addition, convergent and criterion validity indicators are assessed through comparison between automated scores and expert evaluations. The obtained results demonstrate high system reliability (Person Reliability > 0.88), satisfactory item fit to the Rasch model, and the absence of systematic bias related to gender or level of academic preparation. The theoretical and practical implications of integrating generative AI into high-stakes assessment systems are discussed. This study contributes to the advancement of psychometric evaluation of AI-based assessment systems and provides a scientific foundation for their further standardization.