Date
Publisher
arXiv
The rapid development of Generative AI is bringing innovative changes to education and assessment. As the prevalence of students utilizing AI for assignments increases, concerns regarding academic integrity and the validity of assessments are growing. This study utilizes the Earth Science I section of the 2025 Korean College Scholastic Ability Test (CSAT) to deeply analyze the multimodal scientific reasoning capabilities and cognitive limitations of state-of-the-art Large Language Models (LLMs), including GPT-4o, Gemini 2.5 Flash, and Gemini 2.5 Pro. Three experimental conditions (full-page input, individual item input, and optimized multimodal input) were designed to evaluate model performance across different data structures. Quantitative results indicated that unstructured inputs led to significant performance degradation due to segmentation and Optical Character Recognition (OCR) failures. Even under optimized conditions, models exhibited fundamental reasoning flaws. Qualitative analysis revealed that "Perception Errors" were dominant, highlighting a "Perception-Cognition Gap" where models failed to interpret symbolic meanings in schematic diagrams despite recognizing visual data. Furthermore, models demonstrated a "Calculation-Conceptualization Discrepancy," successfully performing calculations while failing to apply the underlying scientific concepts, and "Process Hallucination," where models skipped visual verification in favor of plausible but unfounded background knowledge. Addressing the challenge of unauthorized AI use in coursework, this study provides actionable cues for designing "AI-resistant questions" that target these specific cognitive vulnerabilities. By exploiting AI's weaknesses, such as the gap between perception and cognition, educators can distinguish genuine student competency from AI-generated responses, thereby ensuring assessment fairness.
Study at a glance
- Question: How do advanced multimodal language models perform and reason on complex science exam questions under different input formats?
- Setting and data: South Korea, 20 questions from the 2025 College Scholastic Ability Test Earth Science I exam.
- Method: Mixed methods model evaluation (quantitative scoring across three input formats and AI-assisted qualitative error analysis).
Why it matters
As students increasingly use artificial intelligence for assignments, concerns regarding academic integrity, evaluation fairness, and the validity of assessments are growing.
Key findings
- Under optimized text-and-image input conditions, Gemini 2.5 Pro scored 68%, while GPT-4o scored 22% and Gemini 2.5 Flash scored 20%.
- Full-page image inputs caused severe performance drops across all models due to item segmentation and text-recognition failures, scoring between 8% and 28%.
- Perception errors accounted for 43.06% of errors under optimized conditions, revealing gaps where models misread data or failed to interpret symbolic diagrams.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
What is the application?
Who is the user?
Who age?
Why use AI?
Study design
