Date
Publisher
arXiv
Timely and high-quality feedback is essential for effective learning in
programming courses; yet, providing such support at scale remains a challenge.
While AI-based systems offer scalable and immediate help, their responses can
occasionally be inaccurate or insufficient. Human instructors, in contrast, may
bring more valuable expertise but are limited in time and availability. To
address these limitations, we present a hybrid help framework that integrates
AI-generated hints with an escalation mechanism, allowing students to request
feedback from instructors when AI support falls short. This design leverages
the strengths of AI for scale and responsiveness while reserving instructor
effort for moments of greatest need. We deployed this tool in a data science
programming course with 82 students. We observe that out of the total 673
AI-generated hints, students rated 146 (22%) as unhelpful. Among those, only 16
(11%) of the cases were escalated to the instructors. A qualitative
investigation of instructor responses showed that those feedback instances were
incorrect or insufficient roughly half of the time. This finding suggests that
when AI support fails, even instructors with expertise may need to pay greater
attention to avoid making mistakes. We will publicly release the tool for
broader adoption and enable further studies in other classrooms. Our work
contributes a practical approach to scaling high-quality support and informs
future efforts to effectively integrate AI and humans in education.
Study at a glance
- Question: How do students and instructors use a hybrid help system that allows escalating unhelpful AI hints to instructors?
- Setting and data: Online Master's data science Python course at the University of Michigan, 82 enrolled students, 673 AI hint requests, and 16 instructor escalations.
- Method: Mixed methods analyzing interaction logs and qualitative coding of student code, AI hints, and instructor feedback.
Why it matters
AI-generated hints offer scalable feedback but can be inaccurate, while human instructors provide expertise and instructor presence but face limited time. Combining both may balance responsiveness, accuracy, and support.
Key findings
- Students requested 673 AI hints and rated 22% as unhelpful, but escalated only 16 of those unhelpful hints to instructors, all for debugging issues.
- While awaiting instructor feedback for an average of 13.5 hours, students continued coding in 87.5% of cases and solved the problem beforehand only 25% of the time.
- Instructor responses were evaluated as high-quality in only 44% of escalations, with low-quality responses especially frequent (86%) when the initial AI hint was incorrect.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
What is the application?
Who is the user?
Who age?
Why use AI?
