Search and Filter

Mathedu: Feedback Generation On Problem-Solving Processes For Mathematical Learning Support

Date

AI Overview

Why it matters

Students increasingly use generative AI to learn math, but models risk validating flawed logic or misinterpreting reasoning. Assessing student problem-solving steps and delivering effective feedback remains under-explored.

Study at a glance
  • Question: How reliably can language and reasoning models grade students' step-by-step math problem-solving processes and generate helpful feedback?
  • Setting and data: 4,048 student problem-solving solutions from 6 university students across 4,500 MathQA questions, annotated with feedback by 3 math education experts.
  • Method: Benchmark evaluation comparing prompting and fine-tuning (LoRA) strategies across three tasks, using automated metrics and human expert review.
  • AI tools: Llama3 8B, Llama3 70B, GPT-3.5, o1-mini, GPT-4, GPT-4o
Key findings
  • Fine-tuning a smaller model notably improved answer correctness and error step identification, but the reasoning model o1-mini achieved the highest overall classification performance.
  • Generated feedback across all models lagged behind teacher feedback, often failing to address underlying misconceptions while proving overly verbose or introducing irrelevant information.
  • Models struggled particularly with calculation errors, frequently missing arithmetic mistakes or mistakenly revising correct steps, and occasionally adopted discouraging or judgmental phrasing.

Summary drafted with AI from the full paper and may contain errors. See the full paper for details.

The increasing reliance on Large Language Models (LLMs) across various domains extends to education, where students progressively use generative AI as a tool for learning. While prior work has examined LLMs' mathematical ability, their reliability in grading authentic student problem-solving processes and delivering effective feedback remains underexplored. This study introduces MathEDU, a dataset consisting of student problem-solving processes in mathematics and corresponding teacher-written feedback. We systematically evaluate the reliability of various models across three hierarchical tasks: answer correctness classification, error identification, and feedback generation. Experimental results show that fine-tuning strategies effectively improve performance in classifying correctness and locating erroneous steps. However, the generated feedback across models shows a considerable gap from teacher-written feedback. Critically, the generated feedback is often verbose and fails to provide targeted explanations for the student's underlying misconceptions. This emphasizes the urgent need for trustworthy and pedagogy-aware AI feedback in education.

Publication Source

Publisher
arXiv