Date
AI Overview
Why it matters
Unrestricted large language model tutors can hallucinate, reveal complete solutions, and foster passive dependence. Novice programming students often lack the expertise to judge whether generated code feedback is valid.
Study at a glance
- Question: Can an explicit verification and pedagogical control pipeline make AI programming tutoring safer and more effective?
- Setting and data: 10 undergraduate programming students; 600 instructor-authored queries; 150 public CS50 Stack Exchange and Stack Overflow questions.
- Method: Model benchmark evaluation and counterbalanced within-subject user study.
- AI tools: EduGuard, Meta-Llama-3.1-8B-Instruct, gpt-4o-mini-2024-07-18, DeBERTa-v3-large-MNLI
Key findings
- On the BILearn-CS benchmark, EduGuard achieved 90.1% correctness, 89.4% grounding, and reduced hallucination to 4.9% and direct-answer leakage to 9.8%.
- In a controlled pilot with 10 undergraduates, EduGuard raised post-test accuracy from 68.4% to 81.2% and lowered overreliance from 38.0% to 17.0% compared to GPT-4o-mini Tutor.
- On the public CS50-Forum dataset, EduGuard maintained 86.7% correctness with 6.8% hallucination and 8.9% answer leakage.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
Generative AI (GenAI) is increasingly used by students for programming explanation, debugging, and assignment support. Yet unrestricted large language model (LLM) tutors can hallucinate, contradict course policy, reveal complete solutions, and foster passive dependence. This paper presents EduGuard, a safe retrieval-augmented generation (RAG) tutoring framework for introductory programming. EduGuard integrates query understanding, instructor-approved course retrieval, pedagogical strategy selection, rubric-aware generation, claim-level verification, and overreliance control. To make evaluation provenance explicit, we construct BILearn-CS, a 600-query instructor-authored, TA-validated benchmark spanning concept questions, debugging cases, misconceptions, assignment-support requests, code-mixed Bangla-English queries, and adversarial direct-answer prompts. Moving beyond a synthetic-only benchmark, we further evaluate on a 150-query public CS50-style course-forum set and run a small controlled pilot with 10 undergraduates using a counterbalanced pre-test/post-test design. Using Meta-Llama-3.1-8B-Instruct as the primary generator, hybrid FAISS/BM25 retrieval, and DeBERTa-v3-large-MNLI as an architecturally separate verifier, EduGuard is compared against strong baselines: GPT-4o-mini Tutor, Llama Socratic Tutor, LPITutor-style RAG, RAG with rubric prompting, and RAG with same-model self-checking. On BILearn-CS, EduGuard attains the best correctness (90.1%), grounding (89.4%), and rubric alignment (90.8%), with the lowest hallucination (4.9%) and direct-answer leakage (9.8%). In the pilot, it raises immediate post-test accuracy from 68.4% to 81.2% and cuts overreliance from 38.0% to 17.0% relative to GPT-4o-mini Tutor. These results suggest safe GenAI tutoring requires not only retrieval or strong prompting, but explicit pedagogical control, evidence verification, and deployment safeguards.
Research Categories
What is the application?
Who is the user?
Who age?
Why use AI?
Methods
Datasets
Publication Source
Publisher
arXiv
