Date
Publisher
arXiv
Small Language Models (SLMs) offer privacy and efficiency for educational deployment, yet their utility depends on reliable multistep reasoning. Existing benchmarks often prioritize final answer accuracy, obscuring 'right answer, wrong procedure' failures that can reinforce student misconceptions. This work investigates SLM physics reasoning reliability, stage wise failure modes, and robustness under paired contextual variants. We introduce Physbench, comprising of 3,162 high school and AP level physics questions derived from OpenStax in a structured reference solution format with Bloom's Taxonomy annotations, plus 2,700 paired culturally contextualized variants. Using P-REFS, a stage wise evaluation rubric, we assess 10 SLMs across 58,000 responses. Results reveal substantial reliability gap: among final answer correct solutions, 75 to 98% contain at least one reasoning error. Failure modes shift with model capability; weaker models fail primarily at interpretation or modeling while stronger models often fail during execution. Paired contextual variations have minimal impact on top models but degrade the performance of mid-tier models. These findings demonstrate that safe educational AI requires evaluation paradigms that prioritize reasoning fidelity over final-answer correctness.
Study at a glance
- Question: How reliably do small AI models reason step-by-step through high school physics problems?
- Setting and data: 3,162 high school and AP physics questions, 2,700 contextual variants, and 58,000 responses from 10 small language models.
- Method: Rubric-based evaluation of step-by-step reasoning errors across small AI models (P-REFS).
Why it matters
Small Language Models offer privacy and efficiency for education, yet their utility depends on reliable multistep reasoning. Existing benchmarks often prioritize final answer accuracy, obscuring 'right answer, wrong procedure' failures that can reinforce student misconceptions.
Key findings
- On high school and AP physics questions, 75% to 98% of small language models' correct final answers contained at least one reasoning error.
- Weaker small models failed mostly when interpreting or modeling problems, while stronger models failed more often during solution execution.
- Changing question contexts had minimal impact on top-performing small models but degraded the performance of mid-tier models.
Summary drafted with AI from the paper’s abstract and may contain errors. See the full paper for details.
What is the application?
Who age?
Why use AI?
Study design
