Date
Publisher
arXiv
Generative artificial intelligence (AI) can produce educational content at scale, including interactive and narrative learning experiences, but technical generation alone is not sufficient: scenarios that are confusing, narratively inconsistent, or unengaging are unlikely to be useful in practice. This paper presents a pilot user-centred evaluation of AI-generated interactive fiction (IF) for educational use in higher education. Using a previously described domain-agnostic pipeline and a shared STEM content base, we generated a controlled pool of scenarios and asked participants (N = 22, STEM higher-education) to play one generated episode and rate it on narrative clarity, story-content coherence, engagement, and length acceptance. A free-text prompt captured open feedback. Narrative clarity and length acceptance were rated positively, engagement sat near the neutral mid-point of the scale, and story-content coherence was the weakest dimension by a clear margin. Qualitative feedback points to quiz integration as the bottleneck. Artificial in-fiction motivation for quiz prompts and abrupt setting changes were reported. Feedback also pointed to missing story-level consequences for wrong answers. From these observations, we derive concrete design implications that can inform larger follow-up studies, including later work on learning effectiveness.
Study at a glance
- Question: How do higher-education learners perceive the clarity, coherence, and engagement of AI-generated interactive educational fiction?
- Setting and data: Online pilot study, April 2026, 22 STEM higher-education participants (students, staff, and recent graduates) from a German university.
- Method: Mixed methods user study combining post-game Likert questionnaires, gameplay telemetry, and inductive qualitative analysis of open-ended feedback.
Why it matters
AI can produce interactive educational narratives at scale, but technical generation alone is insufficient if scenarios are confusing, inconsistent, or unengaging for learners.
Key findings
- Participants rated story-content coherence as the weakest dimension (mean 2.92 out of 5), while rating narrative clarity (4.11) and length acceptance (4.14) positively.
- Engagement hovered near the neutral midpoint (mean 3.08 out of 5), with the narrative absorption item scoring lowest at 2.64.
- Qualitative feedback revealed that the main bottleneck was poor quiz integration, specifically artificial motivations for quiz prompts and abrupt scene transitions.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
What is the application?
Who age?
Why use AI?
Study design
