Date
Publisher
arXiv
Timely and high-quality feedback is essential for effective learning in programming courses; yet, providing such support at scale remains a challenge. While AI-based systems offer scalable and immediate help, their responses can occasionally be inaccurate or insufficient. Human instructors, in contrast, may bring more valuable expertise but are limited in time and availability. To address these limitations, we present a hybrid help framework that integrates AI-generated hints with an escalation mechanism, allowing students to request feedback from instructors when AI support falls short. This design leverages the strengths of AI for scale and responsiveness while reserving instructor effort for moments of greatest need. We deployed this tool in a data science programming course with 82 students. We observe that out of the total 673 AI-generated hints, students rated 146 (22%) as unhelpful. Among those, only 16 (11%) of the cases were escalated to the instructors. A qualitative investigation of instructor responses showed that those feedback instances were incorrect or insufficient roughly half of the time. This finding suggests that when AI support fails, even instructors with expertise may need to pay greater attention to avoid making mistakes. We will publicly release the tool for broader adoption and enable further studies in other classrooms. Our work contributes a practical approach to scaling high-quality support and informs future efforts to effectively integrate AI and humans in education.
Study at a glance
- Question: How do students and instructors use a hybrid help system that allows escalating unhelpful AI hints to instructors?
- Setting and data: Online introductory Python course for data science, Master's program at University of Michigan, 82 enrolled students, 673 hint requests across four assignments.
- Method: Mixed methods system deployment analyzing log data, response times, student behaviors, and qualitative coding of help requests and feedback.
Why it matters
AI-generated hints offer scalable feedback but can be inaccurate or lack human nuance, while instructors have limited time. Combining both approaches could leverage AI scale while preserving instructor expertise for moments of greatest need.
Key findings
- Students requested 673 AI hints and rated 22% as unhelpful, but escalated only 16 of those instances to human instructors.
- Instructors took an average of 13.5 hours to respond, yet students stayed engaged by continuing to code or watch lecture videos.
- Instructor responses were evaluated as high-quality in only 44% of escalated cases, frequently struggling when an AI hint had already been incorrect.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
What is the application?
Who age?
Why use AI?
Study design
