Date
Publisher
arXiv
The automated extraction of structured questions from paper-based mathematics exams is fundamental to intelligent education, yet remains challenging in real-world settings due to severe visual noise. Existing benchmarks mainly focus on clean documents or generic layout analysis, overlooking both the structural integrity of mathematical problems and the ability of models to actively reject incomplete inputs. We introduce MathDoc, the first benchmark for document-level information extraction from authentic high school mathematics exam papers. MathDoc contains \textbf{3,609} carefully curated questions with real-world artifacts and explicitly includes unrecognizable samples to evaluate active refusal behavior. We propose a multi-dimensional evaluation framework covering stem accuracy, visual similarity, and refusal capability. Experiments on SOTA MLLMs, including Qwen3-VL and Gemini-2.5-Pro, show that although end-to-end models achieve strong extraction performance, they consistently fail to refuse illegible inputs, instead producing confident but invalid outputs. These results highlight a critical gap in current MLLMs and establish MathDoc as a benchmark for assessing model reliability under degraded document conditions. Our project repository is available at \href{https://github.com/winnk123/papers/tree/master}{GitHub repository}
Study at a glance
- Question: How accurately do multimodal models extract structured exam questions and actively refuse damaged or illegible inputs?
- Setting and data: Authentic high school mathematics exam papers, 460 images containing 3,609 curated questions with real-world noise artifacts.
- Method: Model and benchmark evaluation comparing extraction accuracy and refusal rates across ten multimodal models.
Why it matters
Extracting structured questions from paper-based exams is essential for intelligent education. However, existing benchmarks evaluate clean documents and assume complete inputs, overlooking real-world visual noise and whether vision-language models can actively reject illegible, incomplete questions.
Key findings
- While leading multimodal models achieved strong question extraction accuracy, they consistently failed to refuse illegible inputs, exhibiting low refusal recall across most models.
- Larger models were more sensitive to missing text, with Qwen3-VL-30B reaching a 50 percent refusal rate at 20 percent information loss compared to 40 percent loss for smaller models.
- Cropping individual questions rather than feeding full exam pages substantially reduced refusal failures and helped models actively decline incomplete inputs.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
What is the application?
Who is the user?
Who age?
Why use AI?
Study design
