Search and Filter

LLM-As-Judge In Education: A Curriculum-Grounded Marking Pipeline

Date

AI Overview

Why it matters

Educational assessment is labor-intensive and suffers from marker variability. Simply prompting language models risks drifting from authorized curriculum standards, lacking traceability, and failing to capture professional judgment.

Study at a glance
  • Question: Can an AI pipeline grounded in official curriculum materials mark exam answers reliably and provide traceable feedback?
  • Setting and data: Australia, 47 economics exam questions with human tutor marks, and deployment logs from 5,700 registered students on the Studitory platform, 2026.
  • Method: System development and model evaluation comparing AI and human marks, followed by an online deployment log analysis.
  • AI tools: gpt-5, gpt-5-nano, Claude Opus 4.5, Qwen3 (32B)
Key findings
  • Evaluators preferred the pipeline's written marking justifications over human tutor justifications in 80.4% of pairwise comparisons, and over standalone AI models in 69.6%.
  • Direct prompting of gpt-5 matched human marks more closely than the structured pipeline, achieving a concordance correlation coefficient of 0.81 compared to 0.72.
  • During real-world deployment on the Studitory platform, users manually overrode only 2.91% of AI-generated marks across 3,166 processed student answers.

Summary drafted with AI from the full paper and may contain errors. See the full paper for details.

Generative AI and large language models (LLMs) are increasingly applied to question generation and automated assessment. However, deploying LLMs in preparation for high-stakes exams requires more than prompt engineering; it demands software pipelines that systematically ground model outputs in authorised curriculum artefacts and marking guidelines issued by education authorities. This paper presents a curriculum-grounded, configurable LLM-as-Judge pipeline for question-level marking, co-developed with an industrial partner, to support exam preparation for university admission. The pipeline identifies the relevant topics, subtopics, and cognitive demand of a question, and assembles verifiable and authorised context to support LLM judgement. Curriculum intent is operationalised through concrete syllabus artefacts, including prescribed verbs and outcomes, performance band descriptors, glossary definitions, and marking-guideline principles. A staged LLM workflow is employed to first generate question-specific rubrics, capturing structured expectations of performance, and then derive and evaluate marking criteria used to allocate marks to student responses. This design improves consistency, transparency, and alignment with official marking practices. Preliminary evaluation shows that the proposed LLM-as-Judge pipeline delivers marking outcomes comparable to human tutors, while yielding justifications that are more traceable to authorised curriculum artefacts and marking standards. The pipeline has also been integrated into an online study platform, where early deployment data provide initial insights into operational usage and manual overrides.