Date
Publisher
arXiv
Automated feedback generation plays a crucial role in enhancing personalized learning experiences in computer science education. Among different types of feedback, next-step hint feedback is particularly important, as it provides students with actionable steps to progress towards solving programming tasks. This study investigates how students interact with an AI-driven next-step hint system in an in-IDE learning environment. We gathered and analyzed a dataset from 34 students solving Kotlin tasks, containing detailed hint interaction logs. We applied process mining techniques and identified 16 common interaction scenarios. Semi-structured interviews with 6 students revealed strategies for managing unhelpful hints, such as adapting partial hints or modifying code to generate variations of the same hint. These findings, combined with our publicly available dataset, offer valuable opportunities for future research and provide key insights into student behavior, helping improve hint design for enhanced learning support.
Study at a glance
- Question: How do students interact with an AI next-step hint system, and how do they handle unhelpful hints?
- Setting and data: 34 first- and second-year Bachelor's computer science students solving Kotlin tasks in an IDE, plus interviews with 6 students.
- Method: Mixed methods combining process mining of IDE interaction logs with semi-structured interviews.
Why it matters
Next-step hints help students make progress in programming tasks, but little is known about how students interact with AI-driven hint systems or manage less useful hints.
Key findings
- Across hint interactions, 28.48% were positive explicit acceptances, 40.00% were neutral, and 27.05% were negative, with most negative sessions caused by internal system errors.
- Students frequently continued using the hint system after errors, waiting an average of over 15 minutes before returning to request another hint.
- To handle unhelpful hints, students selectively applied only parts of suggestions or modified their code to generate multiple variations of hints that they combined into working solutions.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
What is the application?
Who is the user?
Who age?
Study design
