Date
Publisher
arXiv
Automated Essay Scoring (AES) plays a crucial role in educational assessment by providing scalable and consistent evaluations of writing tasks. However, traditional AES systems face three major challenges: (1) reliance on handcrafted features that limit generalizability, (2) difficulty in capturing fine-grained traits like coherence and argumentation, and (3) inability to handle multimodal contexts. In the era of Multimodal Large Language Models (MLLMs), we propose EssayJudge, the first multimodal benchmark to evaluate AES capabilities across lexical-, sentence-, and discourse-level traits. By leveraging MLLMs' strengths in trait-specific scoring and multimodal context understanding, EssayJudge aims to offer precise, context-rich evaluations without manual feature engineering, addressing longstanding AES limitations. Our experiments with 18 representative MLLMs reveal gaps in AES performance compared to human evaluation, particularly in discourse-level traits, highlighting the need for further advancements in MLLM-based AES research. Our dataset and code will be available upon acceptance.
Study at a glance
- Question: How well can multimodal artificial intelligence models evaluate essays compared to human scorers?
- Setting and data: Not stated in the abstract.
- Method: Tests how well 18 multimodal artificial intelligence models score essays compared with human evaluations.
Why it matters
While automated essay scoring provides scalable writing evaluations, traditional systems struggle to handle multimodal contexts, capture traits like coherence and argumentation, and generalize. Benchmarks are needed to evaluate whether multimodal large language models can overcome these longstanding limitations.
Key findings
- Across 18 evaluated models, multimodal AI systems showed clear performance gaps compared to human evaluators when scoring essays.
- The performance gap between multimodal AI models and human graders was especially large for discourse-level writing traits.
- Traditional automated essay scoring systems fail to handle multimodal contexts and struggle to evaluate fine-grained traits like coherence and argumentation.
Summary drafted with AI from the paper’s abstract and may contain errors. See the full paper for details.
What is the application?
Who age?
Why use AI?
Study design
