Date
AI Overview
Why it matters
Generative AI introduces challenges because educators often see only final artefacts, leaving students' prompting, evaluation, and decision-making invisible during learning.
Study at a glance
- Question: How can structured reflective journaling make student decision-making and evaluation during generative AI use visible?
- Setting and data: University of Sydney, first-year software engineering course, 10 weeks, 27 students with 190 journals and 608 interactions.
- Method: Mixed methods combining descriptive aggregation of journal data with qualitative review of student reflections
- AI tools: ChatGPT, Microsoft Copilot, Cogniti.ai
Key findings
- Retained students reported high perceived utility of AI assistance (mean 4.42/5) and high confidence in evaluating AI-generated outputs (mean 4.52/5).
- Critical learning moments frequently occurred when students evaluated contextual suitability, feasibility constraints, and requirements alignment rather than detecting obvious errors or hallucinations.
- Interactions that involved verification were associated with higher mean self-reported confidence ratings than those without verification.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
Generative AI (GenAI) is increasingly embedded in software engineering education, supporting activities such as requirements development, design exploration, documentation, and prototyping. However, educators often have visibility only into final artefacts, with limited insight into how students evaluate, verify, and refine AI-generated outputs during the learning process. This creates challenges for assessing evaluative judgement and responsible AI-assisted practice. This paper introduces the AI Journal, a structured reflection framework designed to make student-GenAI interaction visible in first-year software engineering education. The framework combines execution tracking, which records prompts, outputs, intent, and interaction context, with cognitive auditing, which captures verification strategies, intervention decisions, confidence judgements, critical learning moments, and reflections on AI-supported work. Deployed in a first-semester software engineering course, the AI Journal enabled visibility into aspects of student learning not observable through artefact-based assessments alone. Preliminary observations suggested variation in verification practices, intervention strategies, and perceptions of AI-supported work. Critical learning moments frequently occurred when students evaluated contextual suitability, feasibility, and requirements alignment rather than identifying obvious errors. The AI Journal demonstrates a practical, lightweight, and model-agnostic approach for making AI-assisted learning processes visible. By foregrounding verification, intervention, and reflection, it shifts attention from product-focused assessment toward evaluative judgement and responsible AI-assisted practice.
Research Categories
What is the application?
Who is the user?
Who age?
Why use AI?
Methods
Publication Source
Publisher
arXiv
