Date
AI Overview
Why it matters
In mathematics education, students frequently distinguish between nearly identical diagrams to solve problems. However, large multimodal models remain insufficiently evaluated on their capacity to reason over multiple, visually similar figure-based answer options.
Study at a glance
- Question: How accurately can large multimodal models solve mathematics problems where multiple answer choices are visually similar diagrams?
- Setting and data: Chinese high school and college entrance exams, 2002–2023, 1,800 K–12 math problems with 8,070 diagrammatic options.
- Method: Benchmark evaluation comparing multimodal models under zero-shot conditions, option shuffling, and visual alignment strategies.
- AI tools: Gemini 2.5 Pro, GPT-4.1, GPT-4o, Seed1.6-Thinking, GLM-4.5V, Qwen2.5-VL-72B-instruct
Key findings
- Multimodal models showed consistent accuracy declines as visual similarity among diagrammatic choices increased, with top models like Gemini 2.5 Pro reaching 80.9% overall accuracy compared to 91.3% for humans.
- Models performed worse when problem stems included images alongside visual options, and an error analysis revealed that image–text misalignment accounted for 36% of mistakes.
- Finetuning models on 500 multi-image chain-of-thought examples with explicit visual-textual anchors boosted Qwen2.5-VL-3B accuracy by 12.6 percentage points.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
Large Multimodal Models have achieved remarkable progress in integrating
vision and language, enabling strong performance across perception, reasoning,
and domain-specific tasks. However, their capacity to reason over multiple,
visually similar inputs remains insufficiently explored. Such fine-grained
comparative reasoning is central to real-world tasks, especially in mathematics
and education, where learners must often distinguish between nearly identical
diagrams to identify correct solutions. To address this gap, we present
VisioMath, a curated benchmark of 1,800 high-quality K-12 mathematics problems
in which all candidate answers are diagrams with subtle visual similarities. A
comprehensive evaluation of state-of-the-art LMMs, covering both leading
closed-source systems and widely adopted open-source models, reveals a
consistent decline in accuracy as inter-image similarity increases. Analysis
indicates that the dominant failure mode stems from image-text misalignment:
rather than grounding reasoning in textual cues, models often resort to shallow
positional heuristics, resulting in systematic errors. We further explore three
alignment-oriented strategies, spanning training-free approaches and
finetuning, and achieve substantial accuracy gains. We hope that VisioMath will
serve as a rigorous benchmark and catalyst for developing LMMs toward deeper
diagram understanding, precise comparative reasoning, and grounded
multi-image-text integration.
Research Categories
Why use AI?
Study design
Publication Source
Publisher
arXiv
