Date
AI Overview
Why it matters
Adopting AI tutors is hindered by a lack of clear understanding of their capabilities and pedagogical usefulness, resulting in a lack of trust among key educational stakeholders.
Study at a glance
- Question: How accurately can a lightweight language model evaluate the pedagogical quality of AI tutor responses to student mistakes?
- Setting and data: 491 middle and primary school math dialogues from MRBench (incorporating MathDial and Bridge datasets), paired with LLM and human responses; 14 evaluation study participants.
- Method: Multi-task learning model development (LoMTL), benchmark evaluation, and user usability testing.
- AI tools: AITutor-EvalKit, LoMTL, google/gemma-2-2b-it, Prometheus-7B-v2.0, GPT-5, GPT-4
Key findings
- The 2-billion-parameter LoMTL model achieved 0.72 accuracy and a 0.60 macro-F1 score on the full test set, outperforming Prometheus2 and rivaling GPT-5.
- LoMTL substantially outperformed GPT-5 in identifying mistakes and providing actionable next steps, though it trailed slightly in locating mistakes and offering guidance.
- In a user evaluation with 14 participants, most perceived LoMTL as more accurate than GPT-5 and Prometheus2, rating the interface as very easy to use.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
We present AITutor-EvalKit, an application that uses language technology to evaluate the pedagogical quality of AI tutors, provides software for demonstration and evaluation, as well as model inspection and data visualization. This tool is aimed at education stakeholders as well as *ACL community at large, as it supports learning and can also be used to collect user feedback and annotation.
Research Categories
What is the application?
Who age?
Why use AI?
Methods
Publication Source
Publisher
arXiv
