Date
Publisher
arXiv
This study investigates the reliability and validity of five advanced Large
Language Models (LLMs), Claude 3.5, DeepSeek v2, Gemini 2.5, GPT-4, and Mistral
24B, for automated essay scoring in a real world higher education context. A
total of 67 Italian-language student essays, written as part of a university
psychology course, were evaluated using a four-criterion rubric (Pertinence,
Coherence, Originality, Feasibility). Each model scored all essays across three
prompt replications to assess intra-model stability. Human-LLM agreement was
consistently low and non-significant (Quadratic Weighted Kappa), and
within-model reliability across replications was similarly weak (median
Kendall's W < 0.30). Systematic scoring divergences emerged, including a
tendency to inflate Coherence and inconsistent handling of context-dependent
dimensions. Inter-model agreement analysis revealed moderate convergence for
Coherence and Originality, but negligible concordance for Pertinence and
Feasibility. Although limited in scope, these findings suggest that current
LLMs may struggle to replicate human judgment in tasks requiring disciplinary
insight and contextual sensitivity. Human oversight remains critical when
evaluating open-ended academic work, particularly in interpretive domains.
Study at a glance
- Question: How closely and consistently do large language models match human grades on authentic university student essays?
- Setting and data: Università Cattolica del Sacro Cuore, Italy; 67 Italian-language psychology master's essays, 2024–2025 academic year.
- Method: Model evaluation comparing five language models against human raters across four rubric criteria over three prompt replications.
Why it matters
Traditional essay grading is time-consuming and prone to evaluator fatigue or bias. While large language models show promise for automated scoring, direct comparisons with human raters in authentic, discipline-specific higher education contexts remain limited.
Key findings
- Human–model scoring agreement was consistently near zero across all models (Quadratic Weighted Kappa between -0.04 and 0.02) and lacked statistical significance.
- Within-model grading stability across identical prompt repetitions was weak, with a median Kendall's W below .30 and no statistically significant concordance.
- Models showed moderate agreement on structural coherence and originality, but negligible concordance on context-dependent criteria like pertinence and feasibility.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
What is the application?
Who is the user?
Who age?
Why use AI?
Study design
