Search and Filter

Interpretability Framework For Llms In Undergraduate Calculus

Date

AI Overview

Why it matters

Measuring only final answer accuracy obscures whether an AI model's mathematical reasoning is pedagogically sound, consistent, or fragile. Educators need transparent ways to interpret reasoning steps and detect errors before deploying AI in learning environments.

Study at a glance
  • Question: How can researchers systematically interpret, evaluate, and trace the step-by-step reasoning quality of language models on undergraduate calculus exams?
  • Setting and data: Undergraduate Calculus I, II, and III courses at a US university, each with about 50 students across three semester exams.
  • Method: Model evaluation combining step-by-step reasoning flow parsing, prompt ablation sensitivity analysis, and faculty rubric grading.
  • AI tools: Gemma 3, LLaMA 2, LLaVA-7B
Key findings
  • Gemma performed competitively across Calculus I–III exams, scoring between 62.9% and 94.2%, and outperformed student cohort averages on several exams.
  • Retrieval augmentation improved scores on lower-level exams but degraded performance on complex Calculus III exams, dropping scores from 74.0% to 52.0% with standard retrieval.
  • Language models frequently generated fluent yet flawed solutions, struggling with chain rules, graphical problems, and simple arithmetic while remaining sensitive to prompt phrasing.

Summary drafted with AI from the full paper and may contain errors. See the full paper for details.

Large Language Models (LLMs) are increasingly being used in education, yet their correctness alone does not capture the quality, reliability, or pedagogical validity of their problem-solving behavior, especially in mathematics, where multistep logic, symbolic reasoning, and conceptual clarity are critical. Conventional evaluation methods largely focus on final answer accuracy and overlook the reasoning process. To address this gap, we introduce a novel interpretability framework for analyzing LLM-generated solutions using undergraduate calculus problems as a representative domain. Our approach combines reasoning flow extraction and decomposing solutions into semantically labeled operations and concepts with prompt ablation analysis to assess input salience and output stability. Using structured metrics such as reasoning complexity, phrase sensitivity, and robustness, we evaluated the model behavior on real Calculus I to III university exams. Our findings revealed that LLMs often produce syntactically fluent yet conceptually flawed solutions, with reasoning patterns sensitive to prompt phrasing and input variation. This framework enables fine-grained diagnosis of reasoning failures, supports curriculum alignment, and informs the design of interpretable AI-assisted feedback tools. This is the first study to offer a structured, quantitative, and pedagogically grounded framework for interpreting LLM reasoning in mathematics education, laying the foundation for the transparent and responsible deployment of AI in STEM learning environments.