Date
Publisher
arXiv
Many AI tutors leverage large language models (LLMs) today. Given that LLMs are opaque black boxes, robust evaluation and live experimentation to measure the impact of every change are essential. We pioneered AI-powered tutoring for K-12 with the launch of Khanmigo (Khan Academy, 2023). We describe the metrics we use to measure AI tutoring quality and student engagement as well as various experiments we have run. We highlight the changes that have moved our metrics, including models, prompting, personalization and agents.
Study at a glance
- Question: How can we evaluate and improve the quality and student engagement of an AI tutor in K-12 education?
- Setting and data: Khan Academy's Khanmigo AI tutoring tool for K-12 education, launched in 2023; participant numbers not stated.
- Method: Description of evaluation metrics and live experiments testing updates to an AI tutor.
Why it matters
Many AI tutors leverage large language models today. Because these models are opaque black boxes, robust evaluation and live experimentation to measure the impact of every change are essential.
Key findings
- In K-12 tutoring with Khanmigo, adjustments to models, prompts, personalization, and automated agents measurably changed tutoring quality and student engagement.
- Live experimentation is essential to measure the impact of software updates because large language model tutors operate as opaque systems.
- Tracking both tutoring quality and student engagement metrics reveals how specific technical updates affect the performance of AI tutors.
Summary drafted with AI from the paper’s abstract and may contain errors. See the full paper for details.
What is the application?
Who is the user?
Why use AI?
Study design
