Date
AI Overview
Why it matters
As large language models are increasingly considered for grading, understanding whether AI marking is valid across different task types is essential to avoid undetected systematic biases that undermine fairness and trust.
Study at a glance
- Question: How validly can large language models mark physics assessments across structured questions, essays, and scientific plots?
- Setting and data: 771 Durham University physics exam questions (2018-2022), 1,151 UK secondary and textbook questions, 55 essay scripts (275 essays), and 1,400 scientific plots.
- Method: Model or benchmark evaluation comparing AI scoring to human markers under blind, solution-provided, false-solution, and anchored prompt conditions.
- AI tools: GPT-5.2, Grok 4.1, Claude Opus 4.5, DeepSeek-V3.2, Gemini 3 Pro, DeepSeek-Reasoner
Key findings
- Large language model marking validity was sharply task-dependent across all models, achieving strong rank-order agreement on structured questions and scientific plots, but near-zero agreement on essays.
- Providing correct solutions reduced marking error on structured questions, whereas providing false solutions degraded absolute accuracy while leaving the models' rank-ordering of student answers intact.
- For essays, providing anchored exemplar essays pulled AI score averages closer to human marks but failed to produce any meaningful rank-order agreement with human graders.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
As large language models (LLMs) are increasingly considered for automated assessment and feedback, understanding when LLM marking is valid is essential. We evaluate LLM-as-a-judge marking across three physics assessment formats - structured questions, written essays, and scientific plots - comparing GPT-5.2, Grok 4.1, Claude Opus 4.5, DeepSeek-V3.2, Gemini Pro 3, and committee aggregations against human markers under blind, solution-provided, false-solution, and anchored conditions. We distinguish absolute accuracy from rank-order agreement, since a marking system can match the distribution of human marks while failing to order responses by quality. Across task types, performance is sharply task-dependent. For blind university exam questions ($n=771$) and secondary and university structured questions ($n=1151$), models show robust rank-order agreement with human markers (Spearman $ρ> 0.6$), with official solutions reducing error and strengthening agreement. False solutions degrade absolute accuracy, showing that models defer to provided references, but leave rank-ordering intact. Essay marking behaves fundamentally differently. Across $n=55$ scripts ($n=275$ essays), blind AI marking is harsher and more variable than human marking and adding a mark scheme does not improve rank-order agreement. Anchored exemplars shift the AI mean close to the human mean and compress variance below the human standard deviation, but rank-order agreement remains near-zero. For code-based plot elements ($n=1400$), models achieve high rank-order agreement ($ρ> 0.84$) with near-linear calibration. Across all task types, validity tracks the structure of the assessment task - the extent to which marks can be mapped to explicit, observable grading features - and the reliability of the human benchmark, rather than raw model capability.
Research Categories
What is the application?
Who is the user?
Who age?
Why use AI?
Study design
AI tools
Datasets
Publication Source
Publisher
arXiv
