Date
Publisher
arXiv
Accurate estimation of item (question or task) difficulty is critical for educational assessment but suffers from the cold start problem. While Large Language Models demonstrate superhuman problem-solving capabilities, it remains an open question whether they can perceive the cognitive struggles of human learners. In this work, we present a large-scale empirical analysis of Human-AI Difficulty Alignment for over 20 models across diverse domains such as medical knowledge and mathematical reasoning. Our findings reveal a systematic misalignment where scaling up model size is not reliably helpful; instead of aligning with humans, models converge toward a shared machine consensus. We observe that high performance often impedes accurate difficulty estimation, as models struggle to simulate the capability limitations of students even when being explicitly prompted to adopt specific proficiency levels. Furthermore, we identify a critical lack of introspection, as models fail to predict their own limitations. These results suggest that general problem-solving capability does not imply an understanding of human cognitive struggles, highlighting the challenge of using current models for automated difficulty prediction.
Study at a glance
- Question: Can large language models accurately estimate question difficulty and perceive the cognitive struggles of human learners?
- Setting and data: Four datasets spanning medical licensing (USMLE), reading comprehension (Cambridge), and college admissions (SAT Reading/Writing, SAT Math) with field-tested student difficulty ratings.
- Method: Benchmark evaluation comparing zero-shot language model difficulty predictions and IRT-derived empirical difficulty to human test-taker performance.
Why it matters
Accurate item difficulty estimation is critical for educational assessment, but newly created questions lack historical response data. Whether general-purpose language models can perceive human learners' cognitive struggles without task-specific fine-tuning remains an open question.
Key findings
- Language models show systematic misalignment with human difficulty across domains, averaging a Spearman correlation below 0.50, while agreeing more strongly with each other than with students.
- Prompting models to adopt student proficiency levels or ensembling predictions fails to reliably fix misalignment, with models altering problem-solving accuracy by typically less than 1 percent.
- Language models exhibit a critical lack of self-awareness regarding their own errors, scoring near random guessing (average AUROC of 0.56) when predicting item failure.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
What is the application?
Who is the user?
Who age?
Why use AI?
Study design
