Date
Publisher
arXiv
Recent advances in AI-assisted education have encouraged the integration of
vision-language models (VLMs) into academic assessment, particularly for tasks
that require both quantitative and qualitative evaluation. However, existing
VLM based approaches struggle with complex educational artifacts, such as
programming tasks with executable components and measurable outputs, that
require structured reasoning and alignment with clearly defined evaluation
criteria. We introduce AGACCI, a multi-agent system that distributes
specialized evaluation roles across collaborative agents to improve accuracy,
interpretability, and consistency in code-oriented assessment. To evaluate the
framework, we collected 360 graduate-level code-based assignments from 60
participants, each annotated by domain experts with binary rubric scores and
qualitative feedback. Experimental results demonstrate that AGACCI outperforms
a single GPT-based baseline in terms of rubric and feedback accuracy,
relevance, consistency, and coherence, while preserving the instructional
intent and evaluative depth of expert assessments. Although performance varies
across task types, AGACCI highlights the potential of multi-agent systems for
scalable and context-aware educational evaluation.
Study at a glance
- Question: Can a system of collaborative AI agents evaluate student coding assignments more accurately than a single AI model?
- Setting and data: 360 graduate-level code-based assignments from 60 participants, annotated by domain experts with rubric scores and qualitative feedback.
- Method: Compares a multi-agent AI grading system against a single AI model baseline using expert-graded assignments.
Why it matters
Existing vision-language models struggle to evaluate complex educational artifacts like programming tasks. These assignments feature executable components and measurable outputs that require structured reasoning and alignment with clearly defined evaluation criteria.
Key findings
- On 360 graduate-level programming assignments, a collaborative multi-agent AI system produced more accurate and consistent rubric scores and feedback than a single AI model.
- The multi-agent AI system preserved the instructional intent and evaluative depth of human expert assessments.
- The AI tool's evaluation performance varied across different types of coding tasks.
Summary drafted with AI from the paper’s abstract and may contain errors. See the full paper for details.
What is the application?
Who is the user?
Who age?
Why use AI?
Study design
