Search and Filter

One Click Away: AI Tutoring with Khanmigo in a Two-Year School Experiment

Date

AI Overview

Why it matters

Generative AI has been promoted as a technology that could transform education by providing every student with a personal tutor, but large-scale experimental evidence on its effectiveness has been lacking.

Study at a glance
  • Question: Does using the AI tutor Khanmigo during remedial math classes improve middle school students' math achievement?
  • Setting and data: 18 Tennessee middle schools, two-year study, students in daily remedial mathematics sessions.
  • Method: Cluster randomized controlled trial
  • AI tools: Khanmigo
Key findings
  • Assignment to Khanmigo raised math achievement by 1.3 national percentile ranks per term, or about 0.06 to 0.08 standard deviations over a school year.
  • The observed gains resemble those achieved from Khan Academy practice without AI assistance.
  • Students used the tutor infrequently and rarely held substantive math dialogues, messaging Khanmigo on only a third of practice days and in 17 percent of error sessions.

Summary drafted with AI from the paper’s abstract and may contain errors. See the full paper for details.

Generative AI has been promoted as the technology that could transform education by providing every student a personal tutor. We provide some of the first large-scale experimental evidence, from a two-year cluster randomized trial in 18 Tennessee middle schools in which randomly assigned students used Khan Academy with its AI tutor, Khanmigo, configured to coach rather than give answers, during existing daily remedial mathematics sessions. Assignment raises math achievement by 1.3 national percentile ranks per term, or about 0.06 to 0.08 standard deviations over a school year; the implied effect of a full year of active participation reaches 0.14 standard deviations. These gains resemble those from Khan Academy practice without AI assistance. One explanation is that students used the tutor infrequently and, when they did, rarely engaged it in substantive mathematical dialogue: 96 percent of students tried Khanmigo at least once, but the median student messaged it on only a third of the days they practiced, and in only 17 percent of the exercise sessions in which they made a mistake. Messages that students did send were mostly bare answers or clicks on suggested prompts. The binding constraint appears to be engagement: realizing the promise of AI tutoring will require getting students to use it, not just giving them access.