Date
Publisher
arXiv
Code review is central to software engineering education but hard to scale in capstone projects due to tight deadlines, uneven peer feedback, and limited prior experience. We investigate an LLM-as-reviewer integrated directly into GitHub pull requests (human-in-the-loop) across two cohorts (more than 100 students, 2023--2024). Using a mixed-methods design -- GitHub data, reflective reports, and a targeted survey -- we examine engagement and responsiveness as behavioral indicators of self-regulated learning processes. Quantitatively, the 2024 cohort produced more iterative activity (1176 vs. 581 PRs), while technical issues observed in 2023 (227 failed AI attempts) dropped to zero after tool and instructional refinements. Despite different adoption levels (93\% vs. 50\% of teams using the tool), responsiveness was stable: 32\% (2023) and 33\% (2024) of successfully AI-reviewed PRs were followed by subsequent commits on the same PR. Qualitatively, students used the LLM's structured comments to focus reviews and discuss code quality, while guidance reduced over-reliance. We contribute: (i) an in-workflow design for an AI reviewer that scaffolds learning while mitigating cognitive offloading; (ii) a repeated cross sectional comparison across two cohorts in authentic settings; (iii) a mixed-methods analysis combining objective GitHub metrics with student self-reports; and (iv) evidence-based pedagogical recommendations for responsible, student-led AI-assisted review.
Study at a glance
- Question: How do software engineering students engage with an AI code reviewer, and how do they perceive its utility?
- Setting and data: University of Melbourne, Master of Information Technology capstone students across two cohorts (80 students in 2023, 26 in 2024), using GitHub repositories, reflections, and surveys.
- Method: Mixed methods repeated cross-sectional study using descriptive analysis of GitHub repository data, thematic analysis of student reflections, and survey responses.
Why it matters
Peer code reviews are difficult to scale in student capstones due to tight deadlines, uneven feedback, and limited student experience. Empirical evidence on deploying AI-assisted review in authentic classrooms to scaffold self-regulated learning remains sparse.
Key findings
- Across both cohorts, student responsiveness was stable, with 32 percent of reviewed pull requests in 2023 and 33 percent in 2024 followed by subsequent code commits.
- Technical friction dropped markedly between cohorts, with failed AI attempts falling from 227 in 2023 to zero in 2024 following instructional and tool guardrail refinements.
- Students reported that the AI reviewer supported skill development, style, and debugging, but remained skeptical of its ability to understand system-level context and complex logic.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
What is the application?
Who is the user?
Who age?
Study design
