Date
Publisher
arXiv
Multimodal AI can read handwritten physics solutions, but high-stakes grading requires agreement with official scores and outcomes. This study evaluated GPT-5.5-based grading on 10364 scanned pages from 520 handwritten submissions by 416 unique candidates or students across three assessments: a national Physics Olympiad theory examination, the final Olympiad selection camp with theory and experiment components, and a university quantum-mechanics examination. Each submission was graded twice by AI using the official rubrics. The second round used revised page-by-page and evidence-location instructions developed after first-round disagreement analysis. During grading, AI did not see official human marks or AI--human comparisons. Total-score correlations with official marks were high (0.91--0.97). For the final Olympiad selection, AI recovered the same five-student team as official grading. The second round improved aggregate question-part agreement, especially where first-round disagreements were larger. The main difficulty remained exact partial-credit grading, especially in experimental work. Reliable AI grading therefore depends on detailed rubrics and should be used as a second reader or audit tool under examiner control.
Study at a glance
- Question: Can multimodal AI reliably grade handwritten physics exams and match high-stakes student selection decisions?
- Setting and data: India; 10,364 scanned pages from 520 handwritten exam submissions across 416 unique candidates in two Physics Olympiad stages and one university exam.
- Method: Model evaluation comparing AI scores against moderated official examiner scores across two grading rounds using detailed rubrics.
Why it matters
High-stakes grading requires agreement with official scores and selection outcomes. It remains unknown whether AI can reliably grade complex handwritten physics responses that involve derivations, diagrams, and experimental data.
Key findings
- AI total scores tracked official examiner scores closely, achieving correlations between 0.91 and 0.97 across exams, and identified the exact same five-student Olympiad team as human examiners.
- A second grading round with page-by-page instructions and clarified rubrics raised exact question-part agreement from 63% to 70% and reduced large score differences.
- Exact partial-credit scoring remained the most challenging task, with AI showing a tendency to award more partial credit than human graders for incomplete or visually plausible work.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
What is the application?
Who is the user?
Who age?
Why use AI?
Study design
