Date
Publisher
arXiv
Teaching Computer Science (CS) students how to comprehend and maintain legacy code bases is a critical challenge in software engineering education. While Generative AI (GenAI) assistants like GitHub Copilot improve task completion speed and correctness, their impact on code understanding remains unclear. We conducted a within-subject study with 15 graduate CS students completing feature implementation tasks with and without Copilot. Despite significant performance improvements, participants showed no overall comprehension improvement ($p=0.59$), revealing a \textit{comprehension-performance decoupling}. Further analysis uncovered a \textit{comprehension trade-off}: performance gains negatively correlated with reverse engineering comprehension ($ρ=-0.57$, $p=0.026$) but showed a positive trend with implementation comprehension ($ρ=0.50$, $p=0.06$). A follow-up behavioral analysis revealed that \textit{how} students used Copilot determined outcomes: Engaging in verification loops in which programmers actively reviewed generated code strongly predicted comprehension ($p<0.001$, $r=0.96$), with high-comprehension participants verifying code 4.7 times more frequently than low-comprehension participants. These findings suggest that GenAI tools do not inherently undermine comprehension; rather, passive consumption patterns do. This suggests a need to alter programming education to teach system-level verification skills, and the need to redesign educational GenAI tools to scaffold active cognitive engagement.
Study at a glance
- Question: How does GitHub Copilot assistance influence programmers' code comprehension and programming behaviors when modifying unfamiliar codebases?
- Setting and data: Online graduate software engineering course at a large research university in the western United States; 15 graduate students (14 Master's, 1 Ph.D.).
- Method: Within-subjects experiment with behavioral video coding and post-task comprehension tests.
Why it matters
Teaching computer science students to comprehend and maintain existing codebases is a critical challenge. Although AI assistants improve programming speed and correctness, their relationship with code comprehension remains unclear.
Key findings
- GitHub Copilot cut completion time by 50.7 percent and increased passed tests by 71.4 percent, but produced no corresponding improvement in overall code comprehension.
- Programming performance gains were not significantly correlated with code comprehension scores, showing that successful task completion did not reliably indicate code understanding.
- Participants with higher comprehension performed write-then-view verification loops 4.7 times more often than lower-comprehension peers, who spent nearly three times longer viewing AI responses.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
What is the application?
Who is the user?
Who age?
Why use AI?
Study design
