Date
Publisher
arXiv
This paper introduces the design and development of a framework that integrates a large language model (LLM) with a retrieval-augmented generation (RAG) approach leveraging both a knowledge graph and user interaction history. The framework is incorporated into a previously developed adaptive learning support system to assess learners' code, generate formative feedback, and recommend exercises. Moerover, this study examines learner preferences across three instructional modes; adaptive, Generative AI (GenAI), and hybrid GenAI-adaptive. An experimental study was conducted to compare the learning performance and perception of the learners, and the effectiveness of these three modes using four key log features derived from 4956 code submissions across all experimental groups. The analysis results show that learners receiving feedback from GenAI modes had significantly more correct code and fewer code submissions missing essential programming logic than those receiving feedback from adaptive mode. In particular, the hybrid GenAI-adaptive mode achieved the highest number of correct submissions and the fewest incorrect or incomplete attempts, outperforming both the adaptive-only and GenAI-only modes. Questionnaire responses further indicated that GenAI-generated feedback was widely perceived as helpful, while all modes were rated positively for ease of use and usefulness. These results suggest that the hybrid GenAI-adaptive mode outperforms the other two modes across all measured log features.
Study at a glance
- Question: How do adaptive, generative AI, and hybrid feedback and recommendation modes affect student programming performance and perceptions?
- Setting and data: 87 first- and second-year IT undergraduates at a university in Thailand, 15-day study, 4,956 code submissions and 3,154 feedback ratings.
- Method: Three-group experimental study analyzing learning log features, Likert-scale perception surveys, and open-ended feedback.
Why it matters
Novice programming learners often struggle with core concepts but lack explanatory feedback during independent practice. Generative AI and adaptive systems offer tailored support, but their combined effects on coding performance and perceptions remain understudied.
Key findings
- Students using the hybrid generative AI-adaptive mode achieved the highest proportion of correct submissions (81.52%) and had significantly fewer errors than the adaptive-only group.
- Learners receiving generative AI feedback submitted significantly fewer attempts lacking essential programming logic compared to those using the adaptive-only mode.
- Most students agreed that the GPT-4 feedback was helpful, but generative AI modes frequently recommended repeated exercises that students had already completed.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
What is the application?
Who is the user?
Who age?
Why use AI?
Study design
