Date
Publisher
arXiv
Code comprehension is essential for brownfield programming tasks, in which developers maintain and enhance legacy code bases. Generative AI (GenAI) coding assistants such as GitHub Copilot have been shown to improve developer productivity, but their impact on code understanding is less clear. We replicate and extend a previous study by exploring both performance and comprehension in GenAI-assisted brownfield programming tasks. In a within-subjects experimental study, 18 computer science graduate students completed feature implementation tasks with and without Copilot. Results show that Copilot significantly reduced task time and increased the number of test cases passed. However, comprehension scores did not differ across conditions, revealing a comprehension-performance gap: participants passed more test cases with Copilot, but did not demonstrate greater understanding of the legacy codebase. Moreover, we failed to find a correlation between comprehension and task performance. These findings suggest that while GenAI tools can accelerate programming progress in a legacy codebase, such progress may come without an improved understanding of that codebase. We consider the implications of these findings for programming education and GenAI tool design.
Study at a glance
- Question: How does generative AI assistance affect students' code comprehension and programming behaviors when modifying unfamiliar codebases?
- Setting and data: 15 computer science graduate students in an online course at a research university in the western United States.
- Method: Within-subjects experiment comparing tasks completed with and without GitHub Copilot, paired with comprehension tests and behavioral video analysis.
Why it matters
Teaching computer science students to comprehend and maintain existing codebases is a critical challenge. Although generative AI assistants can improve task completion speed and correctness, their relationship with code comprehension remains unclear.
Key findings
- Although Copilot cut completion time by 50.7 percent and increased passed tests by 71.4 percent, participants showed no corresponding improvement in overall code comprehension.
- Performance improvements with Copilot were not reliably associated with comprehension gains, indicating that successful task completion did not reliably reflect code understanding.
- Participants with higher comprehension engaged in verification loops 4.7 times more frequently than lower-comprehension peers, who spent nearly three times longer viewing Copilot responses.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
What is the application?
Who is the user?
Who age?
Why use AI?
Study design
