Date
Publisher
arXiv
Scientific sketches (e.g., models) offer a powerful lens into students'
conceptual understanding, yet AI-powered automated assessment of such
free-form, visually diverse artifacts remains a critical challenge. Existing
solutions often treat sketch evaluation as either an image classification task
or monolithic vision-language models, which lack interpretability, pedagogical
alignment, and adaptability across cognitive levels. To address these
limitations, we present SketchMind, a cognitively grounded, multi-agent
framework for evaluating and improving student-drawn scientific sketches.
SketchMind comprises modular agents responsible for rubric parsing, sketch
perception, cognitive alignment, and iterative feedback with sketch
modification, enabling personalized and transparent evaluation. We evaluate
SketchMind on a curated dataset of 3,575 student-generated sketches across six
science assessment items with different highest order of Bloom's level that
require students to draw models to explain phenomena. Compared to baseline
GPT-4o performance without SRG (average accuracy: 55.6%), and with SRG
integration achieves 77.1% average accuracy (+21.4% average absolute gain). We
also demonstrate that multi-agent orchestration with SRG enhances SketchMind
performance, for example, GPT-4.1 gains an average 8.9% increase in sketch
prediction accuracy, outperforming single-agent pipelines across all items.
Human evaluators rated the feedback and co-created sketches generated by
\textsc{SketchMind} with GPT-4.1, which achieved an average of 4.1 out of 5,
significantly higher than those of baseline models (e.g., 2.3 for GPT-4o).
Experts noted the system's potential to meaningfully support conceptual growth
through guided revision. Our code and (pending approval) dataset will be
released to support reproducibility and future research in AI-driven education.
Study at a glance
- Question: Can a multi-agent AI framework improve the accuracy and pedagogical quality of evaluating student-drawn science models?
- Setting and data: 3,575 student-generated sketches across six science assessment items aligned with NGSS standards; four expert science educators evaluated feedback on 890 sketches.
- Method: Multi-agent system development, model evaluation against human-expert scores, and double-blind expert rating of model feedback.
Why it matters
Assessing student-drawn scientific sketches remains challenging because automated systems often lack interpretability and pedagogical alignment across different cognitive levels.
Key findings
- Integrating Sketch Reasoning Graphs improved GPT-4o's sketch evaluation accuracy from 55.6% to 77.1%, while GPT-4.1 achieved the highest accuracy at 90.2%.
- Modular multi-agent orchestration consistently outperformed single-agent pipelines across all test items, boosting GPT-4.1 accuracy from 82.8% to 90.2% with graph guidance.
- Expert educators rated SKETCHMIND feedback and modified sketches generated with GPT-4.1 at 4.1 out of 5, compared to 2.3 out of 5 for baseline GPT-4o.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
What is the application?
Who is the user?
Study design
