Date
Publisher
arXiv
Generative AI is becoming part of academic writing, but its educational value depends on how control is shared between learner and system. This study examined an agency gap: performance differences that may arise when AI agent initiative is misaligned with learners' generative AI literacy. Seventy-nine medical and nursing students completed two multimodal analytical writing tasks using healthcare simulation data visualisations. They were randomly assigned to a reactive agent that responded only when prompted or a proactive agent that provided sequenced questions and feedback. Generative AI literacy was measured using the validated 20-item Generative AI Literacy Assessment Test (GLAT). Epistemic network analysis showed that proactive interaction created stronger links among conceptual reasoning, evidence use, and constructive engagement, whereas reactive interaction was more factual and procedural. Ordinal regression showed that generative AI literacy predicted immediate independent writing performance after support was removed, particularly for visual data integration, critical thinking, and overall quality. Condition-specific mediation estimates showed a literacy-performance association in the reactive condition but not in the proactive condition; however, the indirect effects and literacy-by-design interactions were not significant. This pattern is consistent with smaller literacy-related performance differences under proactive scaffolding, but it does not establish a compensatory causal effect. Learner reflections indicated that effective AI writing support requires contextual feedback, dialogic scaffolding, and calibration of initiative to learner needs and task complexity. These findings position interaction design as a potential mechanism for supporting equitable and agency-supportive educational AI agents.
Study at a glance
- Question: How do reactive and proactive AI agents shape students' reasoning and subsequent independent writing performance?
- Setting and data: 79 medical and nursing university students or recent graduates recruited globally via Prolific Academic.
- Method: Randomized controlled trial with mixed methods (epistemic network analysis, ordinal logistic regression, mediation analysis, and qualitative thematic analysis).
Why it matters
The educational value of generative AI depends on how control is shared between learner and system. Little empirical evidence shows how reactive versus proactive agent designs affect students' reasoning, feedback use, and independent performance.
Key findings
- Proactive AI dialogues fostered stronger connections among conceptual reasoning, evidence use, and constructive engagement, whereas reactive interactions were more factual and procedural.
- Higher generative AI literacy predicted 11% to 15% higher odds of better independent performance in visual data integration, critical thinking, and overall writing quality after AI support was removed.
- Neither proactive nor reactive agent design had a significant direct effect on independent writing scores, and agent design did not statistically moderate the literacy-performance relationship.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
What is the application?
Who is the user?
Who age?
Why use AI?
Study design
