Date
Publisher
arXiv
We present MSA-MATHEVAL, our submission to the BEA 2025 Shared Task on evaluating AI tutor responses across four instructional dimensions: Mistake Identification, Mistake Location, Providing Guidance, and Actionability. Our approach uses a unified training pipeline to fine-tune a single instruction-tuned language model across all tracks, without any task-specific architecture modifications. To improve prediction reliability, we introduce a disagreement-aware ensemble inference strategy that enhances coverage of minority labels. Our system achieves strong performance across all tracks, ranking 1st in Providing Guidance, 3rd in Actionability, and 4th in both Mistake Identification and Mistake Location. These results demonstrate the effectiveness of scalable instruction tuning and disagreement-driven modeling for robust, multi-dimensional evaluation of LLMs as educational tutors.
Study at a glance
- Question: How accurately can an AI system evaluate the quality and helpfulness of AI math tutor responses?
- Setting and data: Not stated in the abstract
- Method: Tests how well a trained language model evaluates AI math tutor responses across four instructional dimensions
Why it matters
Evaluating AI math tutors requires assessing responses across four instructional dimensions: mistake identification, mistake location, providing guidance, and actionability. Effective evaluation also requires improving prediction reliability and the coverage of minority labels.
Key findings
- In evaluating AI math tutors, an automated assessment system ranked first in judging how well tutors provide instructional guidance.
- The system placed third in assessing response actionability and fourth in evaluating whether tutors correctly identified and located mistakes.
- A disagreement-aware strategy was linked to greater coverage of rarely seen feedback categories and more reliable tutor evaluations.
Summary drafted with AI from the paper’s abstract and may contain errors. See the full paper for details.
What is the application?
Why use AI?
Study design
