Date
AI Overview
Why it matters
Using AI models to assess student work could offer fast, scalable feedback, but automated grading risks undetected systematic biases that undermine fairness. Understanding whether AI marking validity is task-specific is essential before models are deployed in education.
Study at a glance
- Question: How valid and reliable are large language models when grading different types of physics assessments?
- Setting and data: UK secondary and university physics questions, student and AI responses, including 1,922 structured questions, 275 essays, and 1,400 scientific plots.
- Method: Model or benchmark evaluation comparing AI scoring to human marks across blind, solution-provided, false-solution, and anchored conditions, supplemented with qualitative disagreement analysis.
- AI tools: GPT-5.2, Claude Opus 4.5, Gemini 3 Pro, DeepSeek-V3.2, Grok 4.1, DeepSeek-Reasoner
Key findings
- AI models showed robust rank-order agreement with human marks on structured questions and code-based plots, but achieved near-zero rank-order agreement when marking short-form physics essays.
- Providing correct reference solutions reduced AI marking error on structured questions, but providing false solutions severely worsened accuracy because models deferred to the incorrect references.
- Adding anchored score examples to essay prompts shifted AI average scores closer to human averages, but it failed to improve the models' near-zero rank-order agreement.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
As large language models (LLMs) are increasingly considered for automated assessment and feedback, understanding when LLM marking is valid is essential. We evaluate LLM-as-a-judge marking across four settings spanning three assessment formats - structured questions, written essays, and scientific plots - comparing GPT-5.2, Grok 4.1, Claude Opus 4.5, DeepSeek-V3.2, Gemini 3 Pro, and committee aggregations against human markers under blind, solution-provided, false-solution, and anchored conditions. We distinguish absolute accuracy from rank-order agreement, since a marking system can match the distribution of human marks while failing to order responses by quality. Across task types, performance is sharply task-dependent. For blind university exam questions ($n=771$) and secondary and university structured questions ($n=1151$), models show robust rank-order agreement with human markers (Spearman $ρ> 0.6$), with official solutions reducing error and strengthening agreement. False solutions degrade absolute accuracy, showing that models defer to provided references, but leave rank-ordering intact. Essay marking behaves fundamentally differently. Across $n=55$ scripts, each containing five essays ($n=275$ essays total), blind AI marking is harsher and more variable than human marking and adding marking guidance does not improve rank-order agreement. Anchored exemplars shift the AI mean close to the human mean and compress variance below the human standard deviation, but rank-order agreement remains near-zero. For code-based plot elements ($n=1400$), models achieve high rank-order agreement ($ρ> 0.83$) with near-linear calibration. LLM marking validity depended strongly on the assessment task, and this held for every contemporary model tested. The results also show that the reliability of the human benchmark constrains the claims that can be made about AI-human agreement.
Research Categories
What is the application?
Who is the user?
Who age?
Why use AI?
Study design
AI tools
Publication Source
Publisher
arXiv
