Search and Filter

Student Engagement with GenAI's Tutoring Feedback: A Mixed Methods Study

Date

AI Overview

Why it matters

Understanding how introductory programming students perceive, process, and act upon immediate generative AI feedback is crucial before concluding on its usefulness or effects on learners.

Study at a glance
  • Question: How do introductory programming students think about and act on different AI-generated tutoring feedback components?
  • Setting and data: Introductory programming course at a German university, summer semester 2025, 20 undergraduate students.
  • Method: Mixed methods study using eye-tracking, think-aloud protocols, screen recordings, log analysis, and a questionnaire.
  • AI tools: Tutor Kai, gpt-4.1-2025-04-14, whisper-large-v3-turbo
Key findings
  • Overall, 82.0% of AI feedback requests resulted in at least one code improvement, with students acting on 52.9% of the read feedback components.
  • When verbalized thoughts showed understanding, 76.2% of related feedback components led to code improvements, with actionable next-step hints most frequently prompting improvements.
  • When students verbalized disagreement or a need for more information, 69.5% resulted in no immediate code edits, usually prompting them to collect another feedback component.

Summary drafted with AI from the full paper and may contain errors. See the full paper for details.

How students utilize immediate tutoring feedback in programming education depends on various factors. Among them are the feedback quality, but also students' engagement, i.e., their perception, interpretation, and use of feedback. However, there is limited research on how students engage with various types of tutoring feedback. For this reason, we developed a learning environment that provides students with Python programming tasks and various types of immediate, AI-generated tutoring feedback. The feedback is displayed within four components. Using a mixed-methods approach (think-aloud study and eye-tracking), we conducted a study with 20 undergraduate students enrolled in an introductory programming course. Our research aims to: (1) identify what students think when they engage with the tutoring feedback components, and (2) explore the relations between the tutoring feedback components, students' visual attention, verbalized thoughts, and their immediate actions as part of the problem-solving process. The analysis of students' thoughts while engaging with 380 feedback components revealed four main themes: students express understanding or disagreement, additional information needed, and students explicitly judge the feedback. Exploring the relations between feedback, students' attention, thoughts, and actions showed a clear relationship. While expressions of understanding were associated with improvements, expressions of disagreement or need for additional information prompted students to collect another feedback component rather than act on the current information. These insights into students' engagement and decision-making processes contribute to an increased understanding of tutoring feedback and how students engage with it. Thereby, this work has implications for tool developers and educators facilitating feedback.