Date
Publisher
arXiv
As large language models (LLMs) are increasingly integrated into educational tools, current evaluations on standardized tests predominantly focus on binary outcome accuracy. Instead, an effective AI tutor must exhibit faithful reasoning, elucidate solution strategies, and diagnose specific human misconceptions. To bridge this gap, we introduce a pedagogical diagnostic framework that models English Standardized Test (EST) problem-solving as a traversal through a cognitive framework. Based on this framework, we present ESTBook, a multimodal benchmark encompassing 10,576 questions and 29 task types across five major exams. Unlike traditional datasets, ESTBook goes beyond data aggregation by enriching questions with formalized reasoning trajectories and distractor rationales that capture specific cognitive traps. Through extensive evaluations, we empirically demonstrate the practical utility of our diagnostic framework, showing that identifying cognitive trajectories facilitates the mitigation of performance gap and improves pedagogical reasoning through guided elicitation.
Study at a glance
- Question: How effectively do large language models execute step-by-step reasoning and resist cognitive traps on standardized test questions?
- Setting and data: 10,576 multimodal objective questions across 29 task types from five standardized exams (SAT, GRE, GMAT, TOEFL, IELTS), plus five human testers.
- Method: Benchmarking multimodal models against formalized step-by-step cognitive trajectories and distractor rationales, with targeted elicitation mitigations.
Why it matters
Raw test accuracy is fundamentally insufficient for deploying AI as intelligent tutors. Educational assistants must exhibit faithful step-by-step reasoning, clearly explain solution strategies, and diagnose specific student misconceptions rather than simply arriving at correct final answers.
Key findings
- Models frequently succeeded at initial problem formulation (up to 97% accuracy) but suffered sharp performance drops during intermediate constraint integration and execution steps.
- Models were highly susceptible to misleading distractor traps featuring partial truths or flawed formulas, even when demonstrating strong generative capabilities on the same tasks.
- Targeted elicitation strategies that enforced intermediate cognitive constraints improved model accuracy, such as raising GPT-5's GMAT Problem Solving performance from 31.1% to 53.6%.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
What is the application?
Who age?
Why use AI?
Study design
