Date
Publisher
arXiv
Despite rapid progress in vision-language and large language models (VLMs and LLMs), their effectiveness for AI-driven educational assessment in real-world, underrepresented classrooms remains largely unexplored. We evaluate state-of-the-art VLMs and LLMs on over 14K handwritten answers from grade-4 classrooms in Indonesia, covering Mathematics and English aligned with the local national curriculum. Unlike prior work on clean digital text, our dataset features naturally curly, diverse handwriting from real classrooms, posing realistic visual and linguistic challenges. Assessment tasks include grading and generating personalized Indonesian feedback guided by rubric-based evaluation. Results show that the VLM struggles with handwriting recognition, causing error propagation in LLM grading, yet LLM feedback remains pedagogically useful despite imperfect visual inputs, revealing limits in personalization and contextual relevance.
Study at a glance
- Question: How accurately can multimodal AI models transcribe, grade, and provide feedback on handwritten student assignments in Indonesian primary classrooms?
- Setting and data: Six primary schools in Indonesia (three rural, three urban), Grade 4 students, 646 handwritten answer sheets containing over 14,000 responses in Mathematics and English.
- Method: Model or benchmark evaluation comparing AI transcription, scoring, and feedback generation against expert teacher benchmarks, including human evaluations of feedback quality.
Why it matters
Most AI educational assessment tools have been developed in Global North contexts using clean digital text, leaving their effectiveness largely unexplored in underrepresented, low-tech classrooms where paper-based assignments and diverse handwriting remain standard.
Key findings
- When using vision-based AI for handwriting transcription, GPT-4o produced scores closest to human teachers on essays, achieving mean absolute errors of 5.6 in English and 1.5 in Math.
- Grading accuracy improved when using human-transcribed text rather than AI handwriting recognition, with DeepSeek-Chat and GPT-4o emerging as the top-performing models across subjects.
- Scoring errors for English were higher for rural students than urban students across all models, whereas differences in Math scoring between rural and urban settings were less pronounced.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
What is the application?
Who age?
Why use AI?
Study design
