Date
Publisher
arXiv
Despite universal GenAI adoption, students cannot distinguish task performance from actual learning and lack skills to leverage AI for learning, leading to worse exam performance when AI use remains unreflective. Yet few interventions teaching students to prompt AI as a tutor rather than solution provider have been validated at scale through randomized controlled trials (RCTs). To bridge this gap, we conducted a semester-long RCT (N=979) with four ICAP framework-based instructional conditions varying in engagement intensity with a pre-test, immediate and delayed post-test and surveys. Mixed methods analysis results showed: (1) All conditions significantly improved prompting skills, with gains increasing progressively from Condition 1 to Condition 4, validating ICAP's cognitive engagement hierarchy; (2) for students with similar pre-test scores, higher learning gain in immediate post-test predict higher final exam score, though no direct between-group differences emerged; (3) Our interventions are suitable and scalable solutions for diverse educational contexts, resources and learners. Together, this study makes empirical and theoretical contributions: (1) theoretically, we provided one of the first large-scale RCTs examining how cognitive engagement shapes learning in prompting literacy and clarifying the relationship between learning-oriented prompting skills and broader academic performance; (2) empirically, we offered timely design guidance for transforming GenAI classroom policies into scalable, actionable prompting literacy instruction to advance learning in the era of Generative AI.
Study at a glance
- Question: Can brief classroom instruction improve and sustain university students' learning-oriented AI prompting?
- Setting and data: Introductory computer science course at University of Toronto Mississauga, Canada, fall 2025, 431 undergraduate students.
- Method: Randomized controlled trial with a six-week delayed follow-up.
Why it matters
Universities urge students to use generative AI for learning rather than direct solutions, but how instructors can effectively translate these broad policies into classroom practice remains largely untested.
Key findings
- All four instructional approaches significantly improved learning-oriented prompting, with gains persisting six weeks later and scaling with engagement level from 8 to 51 percentage points immediately.
- Students across all conditions self-reported sustained voluntary use of learning-oriented prompting exceeding once per week, and their confidence grew alongside their prompting skills.
- Final examination performance did not differ significantly between conditions, indicating that gains in prompting skills did not directly translate to higher course achievement.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
What is the application?
Who is the user?
Who age?
Why use AI?
Study design
