Date
Publisher
arXiv
Introductory Software Engineering (SE) courses face rapidly increasing student enrollment numbers, participants with diverse backgrounds and the influence of Generative AI (GenAI) solutions. High teacher-to-student ratios often challenge providing timely, high-quality, and personalized feedback a significant challenge for educators. To address these challenges, we introduce NAILA, a tool that provides 24/7 autonomous feedback for student exercises. Utilizing GenAI in the form of modern LLMs, NAILA processes student solutions provided in open document formats, evaluating them against teacher-defined model solutions through specialized prompt templates. We conducted an empirical study involving 900+ active students at the University of Duisburg-Essen to assess four main research questions investigating (1) the underlying motivations that drive students to either adopt or reject NAILA, (2) user acceptance by measuring perceived usefulness and ease of use alongside subjective learning progress, (3) how often and how consistently students engage with NAILA, and (4) how using NAILA to receive AI feedback impacts on academic performance compared to human feedback.
Study at a glance
- Question: How does using an autonomous AI feedback tool impact undergraduate software engineering students' engagement and exam performance?
- Setting and data: University of Duisburg-Essen, Germany; winter semester 2025/26; 843 active undergraduate software engineering students.
- Method: Multiple linear regression analysis (Dose-Response Model) and student surveys based on the Technology Acceptance Model.
Why it matters
Rising undergraduate enrollments, diverse student backgrounds, and high teacher-to-student ratios make delivering timely, high-quality, and personalized feedback difficult for educators.
Key findings
- Using the tool to confirm high mastery on initial submissions was associated with significantly higher final exam scores (beta = 3.59).
- Relative learning gains achieved through repeated, remedial submissions to the tool did not have a statistically significant relationship with final exam performance.
- Among 165 survey respondents, 60.9% reported using the tool for exam preparation, while common reasons for non-use included non-participation and preferring human feedback.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
What is the application?
Who is the user?
Who age?
Why use AI?
Study design
