Date
Publisher
arXiv
Automated Essay Scoring (AES) is crucial for modern education, particularly with the increasing prevalence of multimodal assessments. However, traditional AES methods struggle with evaluation generalizability and multimodal perception, while even recent Multimodal Large Language Model (MLLM)-based approaches can produce hallucinated justifications and scores misaligned with human judgment. To address the limitations, we introduce CAFES, the first collaborative multi-agent framework specifically designed for AES. It orchestrates three specialized agents: an Initial Scorer for rapid, trait-specific evaluations; a Feedback Pool Manager to aggregate detailed, evidence-grounded strengths; and a Reflective Scorer that iteratively refines scores based on this feedback to enhance human alignment. Extensive experiments, using state-of-the-art MLLMs, achieve an average relative improvement of 21% in Quadratic Weighted Kappa (QWK) against ground truth, especially for grammatical and lexical diversity. Our proposed CAFES framework paves the way for an intelligent multimodal AES system. The code will be available upon acceptance.
Study at a glance
- Question: Can a team of collaborative AI agents score multimodal essays more accurately and align better with human judgment?
- Setting and data: Not stated in the abstract
- Method: Tests how well a collaborative multi-agent AI framework scores multimodal essays compared with ground-truth evaluations
Why it matters
Automated essay scoring is crucial for modern education as multimodal assessments grow. However, traditional scoring methods struggle with multimodal perception, while newer multimodal large language models can produce hallucinated justifications and scores misaligned with human judgment.
Key findings
- A collaborative multi-agent AI system improved agreement with human essay scores by an average of 21% compared to standard advanced models.
- The AI system demonstrated its largest scoring improvements when evaluating grammatical accuracy and vocabulary diversity in essays.
- Standard multimodal AI models can produce made-up explanations and assign essay scores that do not align with human judgment.
Summary drafted with AI from the paper’s abstract and may contain errors. See the full paper for details.
What is the application?
Who age?
Why use AI?
Study design
