Date
Publisher
arXiv
Despite rapid progress in vision-language and large language models (VLMs and
LLMs), their effectiveness for AI-driven educational assessment in real-world,
underrepresented classrooms remains largely unexplored. We evaluate
state-of-the-art VLMs and LLMs on over 14K handwritten answers from grade-4
classrooms in Indonesia, covering Mathematics and English aligned with the
local national curriculum. Unlike prior work on clean digital text, our dataset
features naturally curly, diverse handwriting from real classrooms, posing
realistic visual and linguistic challenges. Assessment tasks include grading
and generating personalized Indonesian feedback guided by rubric-based
evaluation. Results show that the VLM struggles with handwriting recognition,
causing error propagation in LLM grading, yet LLM feedback remains
pedagogically useful despite imperfect visual inputs, revealing limits in
personalization and contextual relevance.
Study at a glance
- Question: How accurately can vision-language and large language models grade handwritten elementary student work and generate feedback?
- Setting and data: Six rural and urban primary schools in Indonesia; 646 Grade 4 handwritten answer sheets comprising over 14,000 student answers in Math and English.
- Method: Model evaluation comparing AI handwriting recognition and scoring against teacher transcriptions and grades, alongside expert human evaluation of feedback quality.
Why it matters
Most AI assessment tools are built for Global North classrooms and digital text. Their effectiveness on real handwritten student work in underrepresented, low-resource settings remains largely unexplored.
Key findings
- Vision-language models struggled with handwriting recognition, which caused error propagation into automated grading, though GPT-4o achieved the lowest scoring errors for essays.
- When human transcriptions were provided instead of model handwriting recognition, grading improved markedly, with DeepSeek-Chat and GPT-4o emerging as the top-performing models.
- Scoring errors were substantially higher on English answers from rural students than urban students, while models produced pedagogically clear feedback that nonetheless struggled with personalization.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
What is the application?
Who is the user?
Who age?
Why use AI?
