Search and Filter

A Benchmark For LLM'S Understanding Of Middle School And High School Science Topics

Date

AI Overview

Why it matters

Most AI benchmarks evaluate general language or advanced science rather than K-12 curricula, leaving educators without evidence to choose appropriate models for science classrooms.

Study at a glance
  • Question: How well do open-weight language models understand middle and high school science standards?
  • Setting and data: Nine open-weight language models evaluated on 1,078 middle school and 1,150 high school science items, plus 240 items reviewed by a former teacher.
  • Method: Benchmark evaluation with psychometric analysis and expert human review
  • AI tools: Nemotron, Gemini, Claude, Llama, Gemma, Mistral
Key findings
  • Most tested models scored above 90% accuracy on middle and high school standards, with larger model size not consistently predicting higher performance.
  • Smaller, locally deployable models like OSS-20b and Nemotron 3 Nano 30b achieved high accuracy, scoring between 94% and 97% on single-shot middle and high school tests.
  • A former science teacher verified that 98.3% of 240 reviewed synthetic benchmark questions aligned with intended science standards, though key practices like data interpretation were missing.

Summary drafted with AI from the full paper and may contain errors. See the full paper for details.

Large language models (LLMs) are increasingly integrated into educational settings, yet educators lack robust, standards-aligned tools to evaluate their effectiveness in K-12 science contexts. Existing benchmarks predominantly assess general language or advanced scientific reasoning, leaving a critical gap in understanding LLMs' performance on content directly relevant to secondary science curricula. To address this gap, we developed a comprehensive NGSS-aligned benchmark for both middle and high school science using a rigorous synthetic data pipeline, multi-judge validation, and item-level psychometric analysis. Nine open-weight LLMs were systematically evaluated using this benchmark, indicating that several smaller, locally deployable models achieved high accuracy across diverse science domains and question types. Our findings indicate that model size did not consistently predict performance, emphasizing the importance of intentional model selection for educational deployment. We then incorporated a human reviewer into the loop, reviewing the items generated by the LLMs for alignment with NGSS standards. The human review indicated that synthetically generated items were not in perfect alignment with the NGSS standards, indicating the benefits of human-in-the-loop item development, the need to explore the intersection of content and pedagogical knowledge, and the need to extend benchmarks to evaluate LLMs' capacity for interactive, evidence-based feedback in educational scenarios.