Date
AI Overview
Why it matters
Evidence on AI-assisted grading in real large-scale Spanish assessments remains limited. Questions persist regarding reliability, validity, and designing operational human oversight workflows to preserve fairness in high-stakes certification testing.
Study at a glance
- Question: Can an AI-assisted scoring system reliably evaluate Spanish student essays while reducing human scoring workload?
- Setting and data: Uruguay national certification exam (Acredita EB) for adults over 21 seeking lower secondary certification; 2024 and 2025 tests, evaluating samples of student essays.
- Method: Prompt engineering and simulation of a human-in-the-loop scoring framework evaluated against human operational scores.
- AI tools: OpenAI GPT-5, LanguageTool
Key findings
- AI model accuracy across rubric items ranged from 60% to 80%, slightly below human inter-rater agreement, with prompt performance generalizing closely across both test editions.
- The AI model showed a systematic conservative bias toward under-grading, producing failing outcomes on passing essays in 15.3% of 2024 cases and 16.5% of 2025 cases.
- Simulations show the human-in-the-loop workflow would reduce the volume of written responses requiring full human scoring by at least 50% while protecting high-stakes decisions.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
The integration of artificial intelligence (AI), particularly large language models (LLMs), into educational assessment has opened new opportunities to enhance the efficiency and scalability of grading processes. This study presents the design and validation of an AI-assisted scoring framework for written responses in a large-scale national assessment. The proposed approach focuses on short written texts of approximately 150-200 words and incorporates a human-in-the-loop strategy to preserve assessment quality while reducing manual workload. The study is grounded in a real operational context, using data from two recent editions of a nationwide test, each comprising approximately 5,000 student responses. We analyze the alignment between AI-generated scores and human raters across multiple rubric dimensions, as well as the impact of the proposed decision flow on pass/fail outcomes. Results show moderate to high agreement between the model and human evaluations in most dimensions, supporting the feasibility of AI assistance in this setting. Moreover, the proposed correction workflow identifies cases where human review is most valuable, enabling a more efficient allocation of expert effort. The findings suggest that AI-assisted scoring can be safely integrated into large-scale assessment processes only when combined with carefully designed human oversight. The paper concludes by discussing practical implications for deployment in national assessment systems and outlining future research directions, including longitudinal monitoring of model-human alignment and the analysis of potential cognitive bias introduced by AI-supported review workflows.
Research Categories
What is the application?
Who is the user?
Who age?
Why use AI?
AI tools
Publication Source
Publisher
arXiv
