Date
Publisher
arXiv
Multimodal large language models (MLLMs), which integrate language and visual
cues for problem-solving, are crucial for advancing artificial general
intelligence (AGI). However, current benchmarks for measuring the intelligence
of MLLMs suffer from limited scale, narrow coverage, and unstructured
knowledge, offering only static and undifferentiated evaluations. To bridge
this gap, we introduce MDK12-Bench, a large-scale multidisciplinary benchmark
built from real-world K-12 exams spanning six disciplines with 141K instances
and 6,225 knowledge points organized in a six-layer taxonomy. Covering five
question formats with difficulty and year annotations, it enables comprehensive
evaluation to capture the extent to which MLLMs perform over four dimensions:
1) difficulty levels, 2) temporal (cross-year) shifts, 3) contextual shifts,
and 4) knowledge-driven reasoning. We propose a novel dynamic evaluation
framework that introduces unfamiliar visual, textual, and question form shifts
to challenge model generalization while improving benchmark objectivity and
longevity by mitigating data contamination. We further evaluate knowledge-point
reference-augmented generation (KP-RAG) to examine the role of knowledge in
problem-solving. Key findings reveal limitations in current MLLMs in multiple
aspects and provide guidance for enhancing model robustness, interpretability,
and AI-assisted education.
Study at a glance
- Question: How well do multimodal large language models perform across disciplines, difficulty levels, contextual shifts, and knowledge-driven reasoning in K-12 exams?
- Setting and data: Real-world K-12 exams spanning six disciplines, 141K instances, and 6,225 knowledge points organized in a six-layer taxonomy.
- Method: Benchmark evaluation using a dynamic testing framework with difficulty levels, temporal shifts, contextual shifts, and reference-augmented generation.
Why it matters
Current benchmarks for multimodal large language models suffer from limited scale, narrow coverage, and unstructured knowledge, offering only static and undifferentiated evaluations.
Key findings
- Evaluations reveal performance limitations in current multimodal large language models across multiple tested dimensions.
- Testing models against unfamiliar visual, textual, and question-format shifts challenges model generalization while addressing benchmark data contamination.
- Evaluating knowledge-point reference-augmented generation demonstrates how referencing structured knowledge points contributes to multimodal problem-solving.
Summary drafted with AI from the paper’s abstract and may contain errors. See the full paper for details.
Who is the user?
Why use AI?
Study design
