Date
AI Overview
Why it matters
Generative AI is widely used in higher education, but evidence on whether off-the-shelf chatbots actually help students learn is scarce.
Study at a glance
- Question: Does access to generative AI help or harm undergraduate student learning during timed academic assignments?
- Setting and data: Middlebury College, Spring 2025, 211 undergraduate students across two in-person sessions.
- Method: In-person randomized controlled trial with follow-up testing and instrumental variables analysis.
- AI tools: ChatGPT, GPT-4o, Claude Fable 5.1, Claude Opus 4.8, GPT-5.5, Pangram 4
Key findings
- AI access increased immediate test scores by 7.2 percentage points (0.28 standard deviations), and a smaller gain of 3.9 percentage points persisted one week later.
- Learning gains persisted for students who used AI as a tutor for augmentation, but faded for those who used it for automated writing.
- AI access shifted study time from writing toward reading and searching, and increased reported learning enjoyment by 11 percent.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
We study whether access to off-the-shelf generative AI changes how much students learn in a timed assignment. In a proctored, in-person randomized experiment, undergraduates study an unfamiliar topic and write an essay with or without AI, then complete unaided assessments immediately and one week later. AI access raises test scores, and a smaller gain persists a week later. Gains remain for students who use AI as a tutor ("augmentation") and fade for students who have AI write for them ("automation"). We document two mechanisms: students shift time from writing toward reading and searching for information, and report greater learning enjoyment.
Research Categories
What is the application?
Who is the user?
Who age?
Why use AI?
Study design
AI tools
Publication Source
Publisher
arXiv
