Date
AI Overview
Why it matters
Holistic automated essay scoring obscures writing dimensions and provides little actionable feedback. Teachers and students need interpretable, rubric-aligned feedback across specific traits, especially for complex argumentative writing.
Study at a glance
- Question: How accurately can small open-source and specialized AI models score specific traits in student argumentative essays?
- Setting and data: 1,783 argumentative essays written by Grade 8 students from Prompt 1 of the ASAP++ dataset.
- Method: Model evaluation comparing structured in-context learning with language models against supervised encoder-based ordinal regression models.
- AI tools: BigBird, Llama-3.1 8B, Gemma-3 12B, Ministral-3 8B, GPT-4o-mini, GPT-5.1
Key findings
- A supervised BigBird model trained with ordinal regression achieved the highest agreement with human raters across all five writing traits.
- The small open-source model Ministral-3 8B performed competitively with much larger proprietary models like GPT-5.1 without task-specific fine-tuning.
- Language models agreed more strongly with human raters on reasoning-focused traits like content and organization than on surface-level traits like conventions.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
Automated Essay Scoring systems have traditionally focused on holistic scores, limiting their pedagogical usefulness, especially in the case of complex essay genres such as argumentative writing. In educational contexts, teachers and learners require interpretable, trait-level feedback that aligns with instructional goals and established rubrics. In this paper, we study trait-based Automatic Argumentative Essay Scoring using two complementary modeling paradigms designed for realistic educational deployment: (1) structured in-context learning with small open-source LLMs, and (2) a supervised, encoder-based BigBird model with a CORAL-style ordinal regression formulation, optimized for long-sequence understanding. We conduct a systematic evaluation on the ASAP++ dataset, which includes essay scores across five quality traits, offering strong coverage of core argumentation dimensions. LLMs are prompted with designed, rubric-aligned in-context examples, along with feedback and confidence requests, while we explicitly model ordinality in scores with the BigBird model via the rank-consistent CORAL framework. Our results show that explicitly modeling score ordinality substantially improves agreement with human raters across all traits, outperforming LLMs and nominal classification and regression-based baselines. This finding reinforces the importance of aligning model objectives with rubric semantics for educational assessment. At the same time, small open-source LLMs achieve a competitive performance without task-specific fine-tuning, particularly for reasoning-oriented traits, while enabling transparent, privacy-preserving, and locally deployable assessment scenarios. Our findings provide methodological, modeling, and practical insights for the design of AI-based educational systems that aim to deliver interpretable, rubric-aligned feedback for argumentative writing.
Research Categories
What is the application?
Who age?
Why use AI?
Study design
AI tools
Publication Source
Publisher
arXiv
