Date
AI Overview
Why it matters
Understanding how introductory programming students perceive, process, and act upon immediate generative AI feedback is crucial before concluding on its usefulness or effects on learners.
Study at a glance
- Question: How do introductory programming students think about and act on different AI-generated tutoring feedback components?
- Setting and data: Introductory programming course at a German university, summer semester 2025, 20 undergraduate students.
- Method: Mixed methods study using eye-tracking, think-aloud protocols, screen recordings, log analysis, and a questionnaire.
- AI tools: Tutor Kai, gpt-4.1-2025-04-14, whisper-large-v3-turbo
Key findings
- Overall, 82.0% of AI feedback requests resulted in at least one code improvement, with students acting on 52.9% of the read feedback components.
- When verbalized thoughts showed understanding, 76.2% of related feedback components led to code improvements, with actionable next-step hints most frequently prompting improvements.
- When students verbalized disagreement or a need for more information, 69.5% resulted in no immediate code edits, usually prompting them to collect another feedback component.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
How students utilize immediate tutoring feedback in programming education
depends on various factors. Among them are the feedback quality, but also
students' engagement, i.e., their perception, interpretation, and use of
feedback. However, there is limited research on how students engage with
various types of tutoring feedback. For this reason, we developed a learning
environment that provides students with Python programming tasks and various
types of immediate, AI-generated tutoring feedback. The feedback is displayed
within four components. Using a mixed-methods approach (think-aloud study and
eye-tracking), we conducted a study with 20 undergraduate students enrolled in
an introductory programming course. Our research aims to: (1) identify what
students think when they engage with the tutoring feedback components, and (2)
explore the relations between the tutoring feedback components, students'
visual attention, verbalized thoughts, and their immediate actions as part of
the problem-solving process. The analysis of students' thoughts while engaging
with 380 feedback components revealed four main themes: students express
understanding or disagreement, additional information needed, and students
explicitly judge the feedback. Exploring the relations between feedback,
students' attention, thoughts, and actions showed a clear relationship. While
expressions of understanding were associated with improvements, expressions of
disagreement or need for additional information prompted students to collect
another feedback component rather than act on the current information. These
insights into students' engagement and decision-making processes contribute to
an increased understanding of tutoring feedback and how students engage with
it. Thereby, this work has implications for tool developers and educators
facilitating feedback.
Research Categories
What is the application?
Who is the user?
Who age?
Why use AI?
Study design
Publication Source
Publisher
arXiv
