Date
Publisher
arXiv
Current Artificial Intelligence (AI)-based tutoring systems (AI tutors) are primarily evaluated based on the pedagogical quality of their feedback messages. While important, pedagogy alone is insufficient because it ignores a critical question: what do students actually do with the feedback they receive? We argue that AI tutor evaluation should be extended with a behavioral dimension grounded in student interaction data, which complements pedagogical assessment. We propose an evaluation framework and apply it to 10,235 code submissions with corresponding AI tutor feedback from an introductory undergraduate programming course to measure whether students act on tutor feedback and whether those actions are applied correctly. Using this framework to compare two deployed AI tutors across different semesters in a large-scale introductory computer science course reveals substantial differences in student engagement patterns that are not captured by pedagogy-only evaluation. Moreover, these engagement-based behavioral signals are more strongly associated with student perception of helpful feedback than pedagogical quality alone, providing a more complete and actionable picture of AI tutor performance.
Study at a glance
- Question: How do student behavioral engagement metrics and pedagogical quality compare in evaluating AI tutor effectiveness?
- Setting and data: Introductory college programming course (CS61A), UC Berkeley, Fall 2024 and Fall 2025, 10,235 code submissions from 1,639 students across 10 problems.
- Method: Observational analysis comparing two AI tutors using LLM-judged feedback uptake and binary logistic regression predicting student helpfulness ratings.
Why it matters
AI tutor evaluations primarily assess the pedagogical quality of feedback in isolation, ignoring whether students actually engage with or correctly apply the advice during problem-solving.
Key findings
- MisconceptionTutor achieved 9 to 21 percentage points higher feedback relevance across assignments than BaselineTutor, distinguishing the tutors more clearly than pedagogical rubrics did.
- Engagement metrics of feedback relevance and successful application were the strongest and most consistent predictors of student-perceived helpfulness across regression models.
- Pedagogically undesired feedback that revealed answers led to higher correct code edits (67.8%–79.4% success) than desired feedback (50.9%–53.0%), suggesting students merely copied solutions.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
What is the application?
Who is the user?
Who age?
Why use AI?
Study design
