Search and Filter

Examining The Usage Of Generative AI Models In Student Learning Activities For Software Programming

Date

AI Overview

Why it matters

Research often focuses on generative AI's task completion rather than whether students achieve meaningful knowledge gains, leaving educators uncertain if learners across different proficiency levels genuinely understand programming concepts when assisted by AI.

Study at a glance
  • Question: How does generative AI assistance compare to conventional online resources in supporting students' programming task performance and knowledge gains?
  • Setting and data: Canadian university, 24 undergraduate students across introductory and intermediate programming courses.
  • Method: Controlled experiment (2x2 between-subjects) with surveys, code testing, and chat transcript analysis.
  • AI tools: ChatGPT, GPT-3.5-turbo
Key findings
  • Beginners with ChatGPT achieved significantly higher task scores (410.7 passed tests) than beginners without it (80.7), but generating complete solutions did not improve conceptual understanding.
  • Intermediate students performed identically on tasks with or without ChatGPT (about 291 passed tests), but those without ChatGPT demonstrated stronger solution and conceptual understanding.
  • Using generative AI for complete solution generation boosted task performance over stepwise generation (average 495 versus 206.7 passed tests), but both over-reliance and minimal use resulted in weaker learning.

Summary drafted with AI from the full paper and may contain errors. See the full paper for details.

The rise of Generative AI (GenAI) tools like ChatGPT has created new opportunities and challenges for computing education. Existing research has primarily focused on GenAI's ability to complete educational tasks and its impact on student performance, often overlooking its effects on knowledge gains. In this study, we investigate how GenAI assistance compares to conventional online resources in supporting knowledge gains across different proficiency levels. We conducted a controlled user experiment with 24 undergraduate students of two different levels of programming experience (beginner, intermediate) to examine how students interact with ChatGPT while solving programming tasks. We analyzed task performance, conceptual understanding, and interaction behaviors. Our findings reveal that generating complete solutions with GenAI significantly improves task performance, especially for beginners, but does not consistently result in knowledge gains. Importantly, usage strategies differ by experience: beginners tend to rely heavily on GenAI toward task completion often without knowledge gain in the process, while intermediates adopt more selective approaches. We find that both over-reliance and minimal use result in weaker knowledge gains overall. Based on our results, we call on students and educators to adopt GenAI as a learning rather than a problem solving tool. Our study highlights the urgent need for guidance when integrating GenAI into programming education to foster deeper understanding.