Date
AI Overview
Why it matters
Large language models have become pervasive in education, yet their impact on learning outcomes remains contentious. Research shows conflicting evidence on whether they help or harm student learning.
Study at a glance
- Question: How do large language models affect student learning, and when do they help or harm learning outcomes?
- Setting and data: 113 graduate students at a Dutch university in Spring 2023; 176 university students in German laboratory experiments in June 2024.
- Method: Two-way fixed effects and instrumental variables panel regression (FE2SLS), combined with two randomized lab experiments.
- AI tools: ChatGPT, gpt-3.5-turbo-0613, gpt-3.5-turbo-0125
Key findings
- In two laboratory experiments, having access to large language models had no statistically significant effect on students' overall learning outcomes.
- Using models to generate exercise solutions increased topic volume but decreased understanding, whereas using models to ask for explanations improved understanding.
- Large language models widened achievement gaps, improving outcomes for students with higher prior knowledge while harming those with lower prior knowledge.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
In this paper, we study how generative AI and specifically large language models (LLMs) impact learning in coding classes. We show across three studies that LLM usage can have positive and negative effects on learning outcomes. Using observational data from university-level programming courses, we establish such effects in the field. We replicate these findings in subsequent experimental studies, which closely resemble typical learning scenarios, to show causality. We find evidence for two contrasting mechanisms that determine the overall effect of LLM usage on learning. Students who use LLMs as personal tutors by conversing about the topic and asking for explanations benefit from usage. However, learning is impaired for students who excessively rely on LLMs to solve practice exercises for them and thus do not invest sufficient own mental effort. Those who never used LLMs before are particularly prone to such adverse behavior. Students without prior domain knowledge gain more from having access to LLMs. Finally, we show that the self-perceived benefits of using LLMs for learning exceed the actual benefits, potentially resulting in an overestimation of one's own abilities. Overall, our findings show promising potential of LLMs as learning support, however also that students have to be very cautious of possible pitfalls.
Research Categories
What is the application?
Who is the user?
Who age?
Why use AI?
AI tools
Publication Source
Publisher
arXiv
