Date
Publisher
arXiv
Students' handwritten math work provides a rich resource for diagnosing cognitive skills, as it captures intermediate reasoning beyond final answers. We investigate how current large language models (LLMs) perform in diagnosing cognitive skills from such work. However, student responses vary widely, often omitting steps or providing only vague, contextually implicit evidence. Despite recent advances in LLMs' multimodal and reasoning capabilities, their performance under such conditions remains underexplored. To address this gap, we constructed MathCog, a benchmark dataset containing 3,036 diagnostic verdicts across 639 student responses to 110 math problems, annotated by teachers using TIMSS-grounded cognitive skill checklists with evidential strength labels (Evident/Vague). Evaluating 18 LLMs, we find that (1) all models underperform (F1 < 0.5) regardless of capability, and (2) performance degrades sharply under vague evidence. Error analysis reveals systematic patterns: models frequently misattribute Vague evidence as Evident, overthink minimal cues, and hallucinate nonexistent evidence. We discuss implications for evidence-aware, teacher-in-the-loop designs for LLM-based cognitive diagnosis in educational settings.
Study at a glance
- Question: How well do large language models diagnose students' cognitive skills from handwritten math work?
- Setting and data: Grades 7–9 math problems from AI-Hub, 639 handwritten student responses across 110 problems, evaluated by 15 middle school teachers.
- Method: Benchmark evaluation comparing 18 models to teacher rubrics, alongside qualitative error analysis.
Why it matters
Handwritten math work captures intermediate reasoning, but student responses often omit steps or provide vague evidence. Evaluating how large language models diagnose cognitive skills under these conditions remains largely unexamined.
Key findings
- All 18 evaluated models showed limited performance in diagnosing cognitive skills, with every model achieving a macro F1 score below 0.5.
- Model performance degraded sharply under vague evidence, with average F1 scores dropping by 0.51 and accuracy dropping by 0.41 compared to evident cases.
- Models frequently over-attributed evidence under vague conditions and asserted strong evidence for incorrect diagnoses, with over-inference being the most common error type.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
What is the application?
Who is the user?
Who age?
Why use AI?
Study design
