Date
AI Overview
Why it matters
Automated programming assessment systems frequently rely on basic compiler or unit test feedback, while comparisons with newer large language model feedback remain largely unexplored regarding their quality and effects on student performance.
Study at a glance
- Question: How do compiler, unit test, and AI-generated feedback affect students' perceptions, problem-solving, and task performance?
- Setting and data: Two Austrian universities, Autumn 2023 to Spring 2024, 212 first-year undergraduate students in introductory programming courses.
- Method: Mixed methods experiment and surveys across three programming exercises, analyzing submission logs and Technology Acceptance Model responses.
- AI tools: GPT-3.5-Turbo
Key findings
- Students receiving AI feedback achieved the highest average correctness score of 65.44, significantly outperforming unit test feedback (57.52) and compiler feedback (41.41).
- Students rated unit test feedback highest for quality and usefulness, with 56% preferring it over AI feedback (26%) and compiler feedback (18%).
- AI feedback led students to submit significantly more attempts (average 8.95) and take longer intervals between submissions (average 113.13 seconds) than other feedback types.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
With the recent rapid increase in digitization across all major industries,
acquiring programming skills has increased the demand for introductory
programming courses. This has further resulted in universities integrating
programming courses into a wide range of curricula, including not only
technical studies but also business and management fields of study.
Consequently, additional resources are needed for teaching, grading, and
tutoring students with diverse educational backgrounds and skills. As part of
this, Automated Programming Assessment Systems (APASs) have emerged, providing
scalable and high-quality assessment systems with efficient evaluation and
instant feedback. Commonly, APASs heavily rely on predefined unit tests for
generating feedback, often limiting the scope and level of detail of feedback
that can be provided to students. With the rise of Large Language Models (LLMs)
in recent years, new opportunities have emerged as these technologies can
enhance feedback quality and personalization.
To investigate how different feedback mechanisms in APASs are perceived by
students, and how effective they are in supporting problem-solving, we have
conducted a large-scale study with over 200 students from two different
universities. Specifically, we compare baseline Compiler Feedback, standard
Unit Test Feedback, and advanced LLM-based Feedback regarding perceived quality
and impact on student performance.
Results indicate that while students rate unit test feedback as the most
helpful, AI-generated feedback leads to significantly better performances.
These findings suggest combining unit tests and AI-driven guidance to optimize
automated feedback mechanisms and improve learning outcomes in programming
education.
Research Categories
What is the application?
Who is the user?
Who age?
Why use AI?
Study design
AI tools
Publication Source
Publisher
arXiv
