Date
AI Overview
Why it matters
Providing timely, rubric-aligned feedback on student-drawn STEM diagrams is challenging at scale. Large multimodal models can parse images but frequently hallucinate, which undermines classroom trust.
Study at a glance
- Question: Can a grammar-in-the-loop framework generate more accurate and actionable feedback on student STEM diagrams than end-to-end models?
- Setting and data: 400 synthetic diagrams (200 free-body diagrams, 200 circuit schematics) with controlled errors, evaluated on 40 test samples per domain.
- Method: Benchmark evaluation comparing a four-stage grammar-constrained pipeline against an end-to-end vision-language model and a vision-only baseline.
- AI tools: Sketch2Feedback, Qwen2-VL-2B, Qwen2-VL-2B-Instruct, LLaVA-1.5-7B
Key findings
- The end-to-end model detected errors better on free-body diagrams (micro-F1 0.471 vs. 0.263), while the grammar pipeline substantially outperformed on circuits (0.329 vs. 0.038).
- The grammar pipeline achieved a perfect actionability rating of 5.0 out of 5 on circuit diagrams using structured feedback templates.
- The grammar pipeline's high circuit hallucination rate of 0.925 stemmed from upstream classical computer vision false alarms rather than model confabulation.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
Providing timely, rubric-aligned feedback on student-drawn diagrams is a persistent challenge in STEM education. While large multimodal models (LMMs) can jointly parse images and generate explanations, their tendency to hallucinate undermines trust in classroom deployments. We present Sketch2Feedback, a grammar-in-the-loop framework that decomposes the problem into four stages -- hybrid perception, symbolic graph construction, constraint checking, and constrained VLM feedback -- so that the language model verbalizes only violations verified by an upstream rule engine. We evaluate on two synthetic micro-benchmarks, FBD-10 (free-body diagrams) and Circuit-10 (circuit schematics), each with 500 images spanning standard and hard noise augmentation tiers, comparing our pipeline against end-to-end LMMs (LLaVA-1.5-7B, Qwen2-VL-7B), a vision-only detector, a YOLOv8-nano learned detector, and an ensemble oracle. On n=100 test samples per benchmark with 95% bootstrap CIs, results are mixed and instructive: Qwen2-VL-7B achieves the highest micro-F1 on both FBDs (0.570) and circuits (0.528), but with extreme hallucination rates (0.78, 0.98). An ensemble oracle that selects the best prediction per sample reaches F1=0.556 with hallucination 0.320 on FBDs, demonstrating exploitable complementarity between grammar and end-to-end approaches. Confidence thresholding at tau=0.7 reduces circuit hallucination from 0.970 to 0.880 with no F1 loss. Hard noise augmentation reveals domain-dependent robustness: FBD detection is resilient while circuit detection degrades sharply. An LLM-as-judge evaluation confirms that the grammar pipeline produces more actionable circuit feedback (4.85/5) than the end-to-end LMM (3.11/5). We release all code, datasets, and evaluation scripts.
Research Categories
What is the application?
Who is the user?
Who age?
Why use AI?
Study design
AI tools
Publication Source
Publisher
arXiv
