Date
Publisher
arXiv
Generative AI can improve students' programming performance, but successful task completion may not reflect what they retain. We examined performance, retention, cognitive load, and ownership in a controlled between-subjects experiment with 59 undergraduate computer science students, 55 were retained for analysis. Participants completed three introductory C programming tasks with access to ChatGPT-4.5 or conventional web search without generative AI. We measured task performance, self-reported mental effort and difficulty, pupillary responses, heart rate variability, and ownership, and assessed cued recall immediately and 48 hours later. ChatGPT-assisted students achieved higher coding scores (89% vs. 69%) but lower recall scores immediately (41% vs. 53%) and after 48 hours (39% vs. 52%). There was no significant difference in the loss of recall information over 48 hours between the groups. Self-reported mental effort increased less across tasks in the ChatGPT condition (Holm-adjusted p = .047), and students attributed less of the submitted code to themselves (45% vs. 81%). Confirmatory physiological tests did not detect significant differences in trajectories between conditions; substantial data loss limits their interpretation. These findings reveal a gap between assisted task performance and subsequent recall and sense of ownership in this setting. They motivate the need for assessment practices and AI learning tools that require students to explain, retrieve, and contribute to the work they submit as active participants in their education.
Study at a glance
- Question: How does using generative AI affect students' programming performance, knowledge retention, cognitive load, and sense of code ownership?
- Setting and data: Single university in Australia, 55 undergraduate introductory computer science students.
- Method: Controlled between-subjects experiment (randomized controlled trial).
Why it matters
Generative AI improves immediate task performance, but educators cannot rely on finished work as a proxy for learning without knowing what students retain or how it alters their cognitive investment.
Key findings
- ChatGPT-assisted students achieved higher coding scores (89% vs. 69%), but scored significantly lower on recall quizzes immediately (41% vs. 53%) and 48 hours later (39% vs. 52%).
- Both groups forgot information at similar rates over 48 hours, indicating that AI-assisted learners retained less knowledge during the initial task rather than forgetting faster afterward.
- AI-assisted students attributed significantly less code ownership to themselves (45% vs. 81%) and reported a smaller increase in mental effort as tasks became harder.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
What is the application?
Who is the user?
Who age?
Why use AI?
Study design
