Search and Filter

From Co-Design To Metacognitive Laziness: Evaluating Generative Ai In Vocational Education

Date

AI Overview

Why it matters

While generative AI can improve teaching workflows, the pathway from improved educator experiences to student learning is complex, especially in vocational education where hands-on skill development is emphasized.

Study at a glance
  • Question: How does co-designing and deploying a generative AI chatbot affect vocational lecturers' teaching and their students' exam performance?
  • Setting and data: Vocational education institute in Singapore; 20 lecturers, 103 AI-assisted students plus comparison cohorts across two technical modules, and 269 chat conversations.
  • Method: Mixed methods design including co-design workshops, pre- and post-surveys, interviews, chat log analysis, and comparative exam performance analysis.
  • AI tools: StudyBuddy
Key findings
  • Using the chatbot did not significantly improve overall student exam scores compared to high-performing baseline or control cohorts in either evaluated module.
  • Chat logs showed signs of metacognitive offloading, with 40.1% of student queries using dependency phrases and 81.9% having no clear relationship to enrolled modules.
  • Stronger students used the tool strategically to verify responses, while weaker students exhibited intensive dependency patterns that may widen achievement gaps.

Summary drafted with AI from the full paper and may contain errors. See the full paper for details.

This study examines the development and deployment of a Generative AI proof-of-concept (POC) designed to support lecturers in a vocational education setting in Singapore. Employing a user-centred, mixed-methods design process, we co-developed an AI chatbot with lecturers to address recurring instructional challenges during exam preparation, specifically managing repetitive questions and scaling feedback delivery. The POC achieved its primary operational goals: lecturers reported streamlined workflows, reduced cognitive load, and observed improved student confidence in navigating course content. However, the deployment yielded unexpected insights into student learning behaviours. Despite enhanced teaching processes, performance data revealed no significant improvement in overall student assessment outcomes. Deep analysis of interaction logs identified concerning patterns, including self-efficacy-driven dependency, "metacognitive laziness" (cognitive offloading), and divergent usage strategies. While high-ability students leveraged the tool for strategic verification, low-ability students frequently used it to bypass cognitive effort, potentially exacerbating performance gaps. These findings suggest that Generative AI's educational influence extends beyond instructional efficiency to shape cognitive engagement, self-regulation, and learner equity. The study raises consequential design questions regarding how AI tools can be engineered to minimise dependency, scaffold metacognitive development, and calibrate support across varying ability levels. We conclude that while Generative AI can substantially enhance the teaching experience, achieving meaningful learning gains requires rigorous attention to learner behaviour and the equitable design of AI-supported environments.