Date
Publisher
arXiv
Automated short-answer scoring lags other LLM applications. We meta-analyze 890 culminating results across a systematic review of LLM short-answer scoring studies, modeling the traditional effect size of Quadratic Weighted Kappa (QWK) with mixed effects metaregression. We quantitatively illustrate that that the level of difficulty for human experts to perform the task of scoring written work of children has no observed statistical effect on LLM performance. Particularly, we show that some scoring tasks measured as the easiest by human scorers were the hardest for LLMs. Whether by poor implementation by thoughtful researchers or patterns traceable to autoregressive training, on average decoder-only architectures underperform encoders by 0.37--a substantial difference in agreement with humans. Additionally, we measure the contributions of various aspects of LLM technology on successful scoring such as tokenizer vocabulary size, which exhibits diminishing returns--potentially due to undertrained tokens. Findings argue for systems design which better anticipates known statistical shortcomings of autoregressive models. Finally, we provide additional experiments to illustrate wording and tokenization sensitivity and bias elicitation in high-stakes education contexts, where LLMs demonstrate racial discrimination. Code and data for this study are available.
Study at a glance
- Question: Why do language models struggle to accurately grade children's short-answer responses?
- Setting and data: Meta-analysis of 890 evaluation results across empirical studies using 10 rubric-scored items from the ASAP-SAS dataset, plus prompt experiments.
- Method: Mixed-effects meta-regression, Bayesian hierarchical modeling, and prompt perturbation experiments.
Why it matters
Automated scoring of student short-answer responses struggles to match human agreement. Language models must align with rubrics and meaning rather than relying on surface patterns, risking inaccurate assessments of student learning.
Key findings
- Decoder-only architectures underperformed bidirectional encoder models by an average agreement gap of 0.37 against human raters.
- Grading tasks requiring deeper reading comprehension reliably lowered model agreement with humans, whereas fact-based science tasks did not.
- Scoring difficulty for human experts had no statistical relationship with language model performance, meaning easy human items were sometimes hardest for models.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
What is the application?
Who is the user?
Why use AI?
Study design
