Date
Publisher
arXiv
Large Language Models (LLMs) are increasingly used as proxy students in the
development of Intelligent Tutoring Systems (ITSs) and in piloting test
questions. However, to what extent these proxy students accurately emulate the
behavior and characteristics of real students remains an open question. To
investigate this, we collected a dataset of 489 items from the National
Assessment of Educational Progress (NAEP), covering mathematics and reading
comprehension in grades 4, 8, and 12. We then apply an Item Response Theory
(IRT) model to position 11 diverse and state-of-the-art LLMs on the same
ability scale as real student populations. Our findings reveal that, without
guidance, strong general-purpose models consistently outperform the average
student at every grade, while weaker or domain-mismatched models may align
incidentally. Using grade-enforcement prompts changes models' performance, but
whether they align with the average grade-level student remains highly model-
and prompt-specific: no evaluated model-prompt pair fits the bill across
subjects and grades, underscoring the need for new training and evaluation
strategies. We conclude by providing guidelines for the selection of viable
proxies based on our findings.
Study at a glance
- Question: Can large language models reliably simulate the abilities of average students in grade-level mathematics and reading?
- Setting and data: 489 multiple-choice items from the National Assessment of Educational Progress (NAEP) for grades 4, 8, and 12 in the United States.
- Method: Item response theory (Rasch model) comparing 11 language models to national student response patterns across generic and grade-enforced prompts.
Why it matters
Simulating students with large language models offers scalable testing of tutors and assessments. However, whether these synthetic proxies accurately emulate the performance and abilities of real students across grade levels remains an open question.
Key findings
- Under standard prompts, strong models consistently outperformed the average student at every grade, while only weaker or domain-mismatched models aligned well incidentally.
- Prompting models to act as average grade-level students altered their performance, but no single model-prompt pair reliably matched average students across grades and subjects.
- Models specialized for education, like LearnLM-1.5-Pro and SocraticLM, did not align better with average student performance than general-purpose language models.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
What is the application?
Why use AI?
Study design
