Search and Filter

When AI Is Wrong On Purpose: How Students Respond To Buggy Genai Code

Date

AI Overview

Why it matters

Generative AI models often produce correct solutions on introductory tasks, reducing learners' incentive to critically inspect generated code. Deliberately injecting realistic bugs may create needed opportunities for students to practice code review and debugging.

Study at a glance
  • Question: How do introductory programming students respond when generative AI models produce natural failures versus deliberately injected bugs?
  • Setting and data: Introductory C programming course, University of Auckland, semester 2 2025; 917 students across 2,636 sessions and 6,071 bug-fixing turns.
  • Method: Mixed methods combining descriptive quantitative analysis of interaction logs with qualitative thematic coding of follow-up prompts and post-task student reflections.
  • AI tools: GPT-4o-mini, Prompt Programming
Key findings
  • Encountering deliberately injected bugs led students to edit code directly more often and resolve errors immediately (89.92% on first turns) compared to natural failures (51.16%).
  • Injected bugs resulted in smaller code edits, whereas natural prompt failures more frequently prompted students to rewrite or reframe their natural language instructions.
  • Most surveyed students agreed that self-written code was easier to understand than AI-generated code (about 64%) and reported that resolving injected bugs provided valuable debugging practice.

Summary drafted with AI from the full paper and may contain errors. See the full paper for details.

As Generative AI (GenAI) becomes increasingly central to software development, CS education is integrating prompt-centered workflows where students describe intended program behavior in natural language to elicit code. However, professional practice requires careful review and verification of GenAI-generated code that may appear correct while containing subtle faults. This creates a challenge for CS1-level activities, where current models often solve tasks correctly and reduce students' incentive to closely inspect generated outputs. We investigate how prompt-centered programming activities can be adapted to better foster these practices. Specifically, we explore an approach where realistic, runnable bugs are injected into otherwise correct solutions, thus requiring students to read and repair generated outputs. We analyzed 2,636 sessions from 917 students, and examined behavior across instances of naturally occurring prompt-related failures and deliberately injected bugs within each session. Our findings show that students responded differently across bug sources. Deliberately injected bugs more often led to direct code edits and higher next-attempt success, suggesting localized repair of near-miss solutions. Prompt-related failures instead more often led students to refine prompts by clarifying constraints, updating function signatures, adding edge cases, or reframing the task. Student reflections reinforce the emphasis on review and repair, describing useful practice in code understanding, code review, and debugging, as well as a more careful verification mindset and greater awareness of GenAI limitations. Ultimately, prompt-related failures and injected bugs together support a pedagogically useful GenAI workflow, where students practice both specification refinement through prompts and debugging through code editing.