Date
Publisher
arXiv
Effective mathematics education requires identifying and responding to students' mistakes. For AI to support pedagogical applications, models must perform well across different levels of student proficiency. Our work provides an extensive, year-long snapshot of how 11 vision-language models (VLMs) perform on DrawEduMath, a QA benchmark involving real students' handwritten, hand-drawn responses to math problems. We find that models' weaknesses concentrate on a core component of math education: student error. All evaluated VLMs underperform when describing work from students who require more pedagogical help, and across all QA, they struggle the most on questions related to assessing student error. Thus, while VLMs may be optimized to be math problem solving experts, our results suggest that they require alternative development incentives to adequately support educational use cases.
Study at a glance
- Question: How accurately do vision-language models describe and evaluate handwritten math work from students with and without errors?
- Setting and data: DrawEduMath dataset of 2,030 handwritten responses to 188 K-12 math problems from ASSISTments, mostly from U.S. Title I public schools.
- Method: Benchmark evaluation using an LLM-judge panel, fixed effects regression, and digital redrawing experiments.
Why it matters
Using AI in classrooms requires models to identify and respond to student mistakes across different proficiency levels, but realistic evaluations of vision-language models on student work are lacking.
Key findings
- Across all 11 evaluated vision-language models, models are consistently worse at describing student work containing math errors than work without errors.
- Vision-language models struggle most on questions assessing student correctness, and 29% to 35% of incorrect descriptions match correct math solutions instead.
- Providing gold text descriptions improves correctness evaluations, but model performance still lags behind other question types and binary error judgments remain difficult.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
What is the application?
Why use AI?
Study design
