Date
Publisher
arXiv
Active learning is widely recognized as an effective approach for improving learning outcomes in introductory programming courses. However, insufficient instructional support often limits students' access to timely, personalized feedback, which is crucial for mastering foundational programming concepts. Although recent advances in AI, particularly large language models, offer scalable opportunities for feedback, concerns about explainability and reliability remain. In this paper, we present an AI-driven classroom assistant that leverages an explainable AI model to analyze student code, map logical errors to instructor-identified misconceptions, and deliver instructor-authored feedback, thereby grounding reliability in instructor-defined pedagogical knowledge. To evaluate the effectiveness of our framework, we conducted an expert evaluation to examine its alignment with instructor-verified feedback and deployed the system in a classroom setting to assess students' perceptions of its usability. Results indicate that the assistant can provide accurate, instructor-verified feedback to students while fostering a positive experience.
Study at a glance
- Question: Can an explainable AI assistant accurately deliver instructor-authored feedback to student code and support active learning?
- Setting and data: FalconCode dataset with 1.5 million Python programs, and 69 introductory Python students at a Southeastern US university.
- Method: System development, offline model evaluation, expert human evaluation of feedback, and a post-activity student survey.
Why it matters
Large class sizes limit instructors from delivering timely, individualized feedback during active learning. While large language models offer scalability, concerns about reliability, explainability, and hallucinations hinder their direct classroom use.
Key findings
- Adding a GPT-4o verification layer reduced the rate of incorrect feedback from 23% to 1.2% in offline evaluations.
- Compared to directly generated GPT-4o feedback, the INSIGHT system achieved higher selectivity (95% versus 53%) and superior pedagogical suitability (100% versus 47%).
- In the classroom deployment, 53.6% of surveyed students expressed interest in using the system again, and 47.9% agreed the feedback was clear.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
What is the application?
Who age?
Why use AI?
Study design
