Date
Publisher
arXiv
Access to academic support is a key determinant of student success, yet students experience it unequally: some readily seek help from lecturers or tutors, while others hesitate due to anxiety, fear of judgement, uncertainty about expectations, or low confidence in their understanding. This may be especially evident in computing education, where programming tasks are cumulative and cognitively demanding. Although students increasingly turn to general-purpose generative AI tools, these can produce responses that are inaccurate, insufficiently contextualised, or misaligned with module expectations. This study presents and evaluates Beacon, a course-specific Retrieval-Augmented Generation (RAG) system providing private, immediate, module-aligned academic support. Grounding responses in approved teaching materials, Beacon was designed to lower barriers to help-seeking while encouraging independent learning. Using a design-based research approach, Beacon was developed iteratively and evaluated via mixed methods, combining questionnaires and semi-structured interviews with students and staff at a Higher Education institution. Students described Beacon's responses as closely aligned with module content and more trustworthy than unrestricted generative AI tools, valuing its use of pseudocode and scaffolded explanations over direct solutions. Although participants remained cautious about trusting AI-generated responses without verification, they viewed the system as a valuable first point of support before consulting lecturers or official resources. The findings suggest that carefully designed course-specific AI systems may reduce barriers to academic support by occupying an intermediary space between independent study and formal support. Rather than replacing educators, educational AI may be most valuable when it broadens access to guidance while preserving the pedagogical role of lecturers.
Study at a glance
- Question: Can a course-specific AI system grounded in teaching materials reduce help-seeking barriers and support student learning?
- Setting and data: Undergraduate computing students and faculty at a British university, including 9 pilot students, 15 evaluation students, and 4 academic staff.
- Method: Design-based research using mixed methods, combining task-based usability testing, pre- and post-questionnaires, and semi-structured interviews.
Why it matters
Students face unequal barriers to academic support due to anxiety, fear of judgement, and low confidence. Unrestricted generative AI tools can provide misaligned or inaccurate guidance, potentially reinforcing these learning disparities.
Key findings
- Most students agreed Beacon made private help-seeking easier (86.7%) and could benefit learners with low confidence (93.3%) or concerns about judgement (80%).
- Nearly all students (93.3%) found responses aligned with module materials, and 86.7% reported that referencing source materials increased their trust in outputs.
- While 80% agreed Beacon helped them learn independently, participants noted frustrations with slow responses, overly long templated answers, and occasional ability to bypass guardrails.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
What is the application?
Who age?
Study design
