Date
Publisher
arXiv
Integration of Large Language Models (LLMs) with Learning Management Systems (LMSs) can enhance task automation and accessibility in education. However, hallucination where LLMs generate inaccurate or misleading information remains a challenge. This study introduces the Dynamic Course Content Integration (DCCI) mechanism, which dynamically retrieves course content from Canvas LMS and structures it within an LLM's context window via prompt engineering, enabling the LLM-powered assistant, Ask ME, to deliver context-aware, curriculum-aligned responses while mitigating hallucinations. A mixed-methods pilot study grounded in Self-Determination Theory (autonomy, competence) and the Technology Acceptance Model (perceived usefulness, ease of use) evaluated DCCI's effectiveness with 120 first-year programming students at Eötvös Loránd University. The course focused on foundational programming patterns in C#, including writing program specifications. We analyzed 14,746 logged interactions and a post-course survey completed by 101 students. User satisfaction was measured via a 5-point Likert scale (turn-level ratings), while the survey assessed usability, engagement, and ethical concerns. Results indicated high satisfaction (mean 4.65/5) and strong recognition of Ask ME's ability to provide timely, contextually relevant answers to administrative and course-related queries. 78.06% agreed that Ask ME's Canvas integration reduced platform switching, improving usability, engagement, comprehension, and topic exploration. Many students reported reduced hesitation to ask questions and increased motivation for self-directed learning, though concerns about over-reliance on AI and reduced student-teacher interaction emerged. This study demonstrates that DCCI enhances LLM reliability, student satisfaction, and engagement in AI-driven educational automation, while highlighting the importance of balancing
Study at a glance
- Question: Can dynamically connecting course materials to an AI assistant inside a learning management system improve student satisfaction and engagement?
- Setting and data: Eötvös Loránd University, first-year programming course, October–December 2024, 120 students, 14,746 logged chatbot interactions, and 101 survey respondents.
- Method: Mixed-methods pilot study evaluating user interaction logs, turn-level ratings, and a post-course survey.
Why it matters
Large language models in education often suffer from hallucinations by generating inaccurate or misleading information, and many standalone AI tools lack direct integration into learning management systems.
Key findings
- Students gave the assistant an average satisfaction rating of 4.65 out of 5, with 90.4% viewing responses as consistent with course materials.
- Integrating the assistant into Canvas reduced platform switching for 78.06% of students and lowered anxiety about asking questions for 72.55%.
- Students voiced concerns about over-reliance on AI, possible inaccuracies, academic dishonesty, and the potential weakening of direct student–teacher interactions.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
What is the application?
Who is the user?
Who age?
Why use AI?
Study design
