Search and Filter

When Shallow Wins: Silent Failures And The Depth-Accuracy Paradox In Latent Reasoning

Date

AI Overview

Why it matters

Mathematical models are widely deployed in education and tutoring, making it essential to validate whether benchmark accuracy reflects reliable internal computation rather than brittle heuristics.

Study at a glance
  • Question: Does correct mathematical problem-solving in models reflect genuine, reliable internal reasoning or superficial shortcuts?
  • Setting and data: 500 multi-step math problems from the GSM8K benchmark.
  • Method: Model evaluation using internal activation analysis, causal noise interventions, and trajectory comparisons.
  • AI tools: Qwen2.5-Math-7B, Qwen2.5-Math-1.5B
Key findings
  • Among correct math predictions by Qwen2.5-Math-7B, only 18.4% used stable, faithful reasoning, while 81.6% emerged through computationally inconsistent pathways.
  • Confident yet incorrect predictions accounted for 8.8% of all model outputs, representing silent failures.
  • Scaling from 1.5B to 7B parameters provided zero accuracy gain on the evaluated subset, with both models achieving an identical 61.0% accuracy.

Summary drafted with AI from the full paper and may contain errors. See the full paper for details.

Mathematical reasoning models are widely deployed in education, automated tutoring, and decision support systems despite exhibiting fundamental computational instabilities. We demonstrate that state-of-the-art models (Qwen2.5-Math-7B) achieve 61% accuracy through a mixture of reliable and unreliable reasoning pathways: 18.4% of correct predictions employ stable, faithful reasoning while 81.6% emerge through computationally inconsistent pathways. Additionally, 8.8% of all predictions are silent failures -- confident yet incorrect outputs. Through comprehensive analysis using novel faithfulness metrics, we reveal: (1) reasoning quality shows weak negative correlation with correctness (r=-0.21, p=0.002), reflecting a binary classification threshold artifact rather than a monotonic inverse relationship; (2) scaling from 1.5B to 7B parameters (4.7x increase) provides zero accuracy benefit on our evaluated subset (6% of GSM8K), requiring validation on the complete benchmark; and (3) latent reasoning employs diverse computational strategies, with ~20% sharing CoT-like patterns. These findings highlight that benchmark accuracy can mask computational unreliability, demanding evaluation reforms measuring stability beyond single-sample metrics.

Publication Source

Publisher
arXiv