Date
Publisher
arXiv
Large Language Models (LLMs) are increasingly used as proxy students in the
development of Intelligent Tutoring Systems (ITSs) and in piloting test
questions. However, to what extent these proxy students accurately emulate the
behavior and characteristics of real students remains an open question. To
investigate this, we collected a dataset of 489 items from the National
Assessment of Educational Progress (NAEP), covering mathematics and reading
comprehension in grades 4, 8, and 12. We then apply an Item Response Theory
(IRT) model to position 11 diverse and state-of-the-art LLMs on the same
ability scale as real student populations. Our findings reveal that, without
guidance, strong general-purpose models consistently outperform the average
student at every grade, while weaker or domain-mismatched models may align
incidentally. Using grade-enforcement prompts changes models' performance, but
whether they align with the average grade-level student remains highly model-
and prompt-specific: no evaluated model-prompt pair fits the bill across
subjects and grades, underscoring the need for new training and evaluation
strategies. We conclude by providing guidelines for the selection of viable
proxies based on our findings.
Study at a glance
- Question: Can AI models accurately simulate real students' abilities in math and reading comprehension?
- Setting and data: 489 math and reading comprehension test items from NAEP for grades 4, 8, and 12, and 11 AI models.
- Method: Compares AI model test performance to real student abilities on a shared scale (Item Response Theory).
Why it matters
Large language models are increasingly used as proxy students to develop intelligent tutoring systems and pilot test questions. However, whether these models accurately emulate the behavior and characteristics of real students remains an open question.
Key findings
- Strong AI models consistently outperformed the average student in grades 4, 8, and 12 math and reading, while weaker models aligned only incidentally.
- Prompting models to match grade levels altered their performance, but alignment with average students remained highly specific to the particular model and prompt.
- No evaluated combination of model and prompt successfully matched average student abilities across all tested subjects and grade levels.
Summary drafted with AI from the paper’s abstract and may contain errors. See the full paper for details.
What is the application?
Why use AI?
Study design
