Search and Filter

When AI Tutors Speak: Evidence From A Randomized Field Experiment

Date

AI Overview

Why it matters

Unguided generative AI invites cognitive offloading and can harm unassisted learning. Evidence is lacking on which tutoring configurations produce learning, and whether speaking with an AI tutor improves outcomes compared to typing.

Study at a glance
  • Question: Does a structured AI tutor improve student learning, and does speaking to it teach better than typing?
  • Setting and data: Online MBA corporate finance course, Boston University, 2026, 86 randomized working-professional students.
  • Method: Randomized controlled trial with a within-student weekly crossover design (voice versus text).
  • AI tools: GPT-5.4, GPT-5, Gemini 2.5 Pro, Claude Opus 5, DeepSeek-V3, ElevenLabs
Key findings
  • Students assigned to the structured AI tutor gained 6.63 more points on a 55-point assessment than peers in the holdout group.
  • Learning gains were statistically equivalent between voice and text tutoring weeks, showing that delivery modality did not affect mastery.
  • Voice interaction doubled dialogue turns per minute and increased question-asking 2.4-fold compared to text, but cost 2.8 times as much to deliver.

Summary drafted with AI from the full paper and may contain errors. See the full paper for details.

Students increasingly study alongside generative artificial intelligence (AI), yet unguided access to fluent answers invites cognitive offloading, and there is little evidence on which configurations of AI tutoring produce learning. Two design margins are usually bundled together: pedagogical structure (how the tutor teaches) and interaction modality (how students talk to it). We separate them. In a preregistered randomized field experiment in a graduate corporate-finance course of an online MBA, we randomized 86 students between a structured tutor grounded in the course materials and a holdout in which consumer AI remained freely available. Within the tutored arm, each student's channel alternated weekly between voice and text, so the modality effect is identified within student. Structure mattered: tutored students gained 6.63 points more than ability-matched peers (p=.007), and the gain was concentrated in written reasoning, where the share of answers reaching relational quality rose from 8% to 49% in the tutored arm against 8% to 27% in the holdout. Modality did not matter for learning. The instructor's own final, on file for all 86 randomized students, shows the same direction (2.6 points of 100, with no difference on a pre-treatment midterm). Voice nearly doubled conversational interaction and cost 2.8 times as much to deliver, yet it produced weekly mastery statistically equivalent to text, even as students came to prefer it. Pedagogical structure shapes what students practice, and modality shapes how they interact with the tutor. Making an AI more humanlike does not by itself make it more educational.