Pedagogy-Driven Evaluation Of Generative Ai-Powered Intelligent Tutoring Systems

Kaushal Kumar Maurya, Mohamed bin Zayed University of Artificial Intelligence

Ekaterina Kochmar, Mohamed bin Zayed University of Artificial Intelligence

Date

10/2025

AI Overview

Why it matters

The true progress and impact of generative AI tutoring systems remain untraceable without reliable, universally accepted, pedagogy-driven evaluation frameworks and benchmarks.

Study at a glance

Question: How should generative AI tutoring systems be evaluated using learning science principles and unified benchmarks?
Setting and data: Review of AI in education research, evaluation benchmarks, and tutoring literature.
Method: Literature review and conceptual analysis

Key findings

Most existing AI tutor evaluations rely on subjective protocols, self-reports, or standard text generation metrics that fail to capture pedagogical value.
Current tutoring benchmarks lack standardized learning science principles, diverse datasets, and multi-turn interaction depth needed for robust assessment.
Evaluation progress is limited by diverse pragmatics, including varying participant roles, unguided versus guided formats, and trade-offs between human judgment and automated scoring.

Summary drafted with AI from the full paper and may contain errors. See the full paper for details.

The interdisciplinary research domain of Artificial Intelligence in Education (AIED) has a long history of developing Intelligent Tutoring Systems (ITSs) by integrating insights from technological advancements, educational theories, and cognitive psychology. The remarkable success of generative AI (GenAI) models has accelerated the development of large language model (LLM)-powered ITSs, which have potential to imitate human-like, pedagogically rich, and cognitively demanding tutoring. However, the progress and impact of these systems remain largely untraceable due to the absence of reliable, universally accepted, and pedagogy-driven evaluation frameworks and benchmarks. Most existing educational dialogue-based ITS evaluations rely on subjective protocols and non-standardized benchmarks, leading to inconsistencies and limited generalizability. In this work, we take a step back from mainstream ITS development and provide comprehensive state-of-the-art evaluation practices, highlighting associated challenges through real-world case studies from careful and caring AIED research. Finally, building on insights from previous interdisciplinary AIED research, we propose three practical, feasible, and theoretically grounded research directions, rooted in learning science principles and aimed at establishing fair, unified, and scalable evaluation methodologies for ITSs.