Date
Publisher
arXiv
Code-generating Artificial Intelligence has gained popularity within both professional and educational programming settings over the past several years. While research and pedagogy are beginning to cope with this change, computing students are left to bear the unforeseen consequences of AI amidst a dearth of empirical evidence about its effects. Though pair programming between students is well studied and known to be beneficial to self-efficacy and academic achievement, it remains underutilized and further threatened by the proposition that AI can replace a human programming partner. In this paper, we present a controlled pair programming study with 22 participants who wrote Python code under time pressure in teams of two and individually with GitHub Copilot for 20 minutes each. They were incentivized by bonus compensation to balance performance with understanding and were retested individually on the programming tasks after a retention interval of one week. Subjective measures of workload and emotion as well as objective measures of performance and learning (retest performance) were collected. Results showed that participants performed significantly better with GitHub Copilot than their human teammate, and several dimensions of their workload were significantly reduced. However, the emotional effect of the human teammate was significantly more positive and arousing as compared to working with Copilot. Furthermore, there was a nonsignificant absolute retest performance reduction in the AI condition and a larger retest performance decrement in the AI condition. We recommend that educators strongly consider revisiting pair programming as an educational tool in addition to embracing modern AI.
Study at a glance
- Question: How do performance, learning retention, workload, and emotions differ between working with a human partner versus an AI assistant?
- Setting and data: Selective research university in eastern United States, 22 undergraduate engineering and data science students with novice or intermediate Python skills.
- Method: Within-subjects lab experiment with counterbalanced pairs, OLS regression with wild cluster bootstrapping, and qualitative exit questionnaires (Mixed methods).
Why it matters
Computing students face unknown consequences from generative AI amidst scarce empirical evidence. AI tools threaten peer pair programming, a practice known to boost self-efficacy and achievement, by promising to replace human partners.
Key findings
- Participants scored roughly 14 points higher out of 100 with GitHub Copilot than with human partners, but did not rate their own performance any higher.
- Mental demand, temporal demand, and effort were significantly lower when working with AI, while frustration showed no significant difference between conditions.
- Working with a human teammate generated significantly more positive and arousing emotions, while learning retention differences after one week were mostly nonsignificant.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
What is the application?
Who is the user?
Who age?
Why use AI?
Study design
