Date
Publisher
arXiv
Generative AI is no longer a peripheral tool in higher education. It is
rapidly evolving into a general-purpose infrastructure that reshapes how
knowledge is generated, mediated, and validated. This paper presents findings
from a controlled experiment evaluating a Socratic AI Tutor, a large language
model designed to scaffold student research question development through
structured dialogue grounded in constructivist theory. Conducted with 65
pre-service teacher students in Germany, the study compares interaction with
the Socratic Tutor to engagement with an uninstructed AI chatbot. Students
using the Socratic Tutor reported significantly greater support for critical,
independent, and reflective thinking, suggesting that dialogic AI can stimulate
metacognitive engagement and challenging recent narratives of de-skilling due
to generative AI usage. These findings serve as a proof of concept for a
broader pedagogical shift: the use of multi-agent systems (MAS) composed of
specialised AI agents. To conceptualise this, we introduce the notion of
orchestrated MAS, modular, pedagogically aligned agent constellations, curated
by educators, that support diverse learning trajectories through differentiated
roles and coordinated interaction. To anchor this shift, we propose an adapted
offer-and-use model, in which students appropriate instructional offers from
these agents. Beyond technical feasibility, we examine system-level
implications for higher education institutions and students, including funding
necessities, changes to faculty roles, curriculars, competencies and assessment
practices. We conclude with a comparative cost-effectiveness analysis
highlighting the scalability of such systems. In sum, this study contributes
both empirical evidence and a conceptual roadmap for hybrid learning ecosystems
that embed human-AI co-agency and pedagogical alignment.
Study at a glance
- Question: Does a Socratic AI tutor better support critical, independent, and reflective thinking than an uninstructed AI chatbot?
- Setting and data: German university, four biology seminars, 65 pre-service teacher education students.
- Method: Controlled experiment (between-subjects design) with multi-level modeling.
Why it matters
Generative AI tools often deliver polished answers with minimal cognitive effort, risking passive consumption and overreliance instead of engaging students in deeper inquiry, critical evaluation, and epistemic agency.
Key findings
- Students using the Socratic tutor reported significantly higher motivation for independent thinking compared to those using the uninstructed chatbot.
- Socratic tutor users perceived significantly greater support for reflective thinking and critical thinking than users of the uninstructed chatbot.
- A five-minute student interaction with the Socratic AI chatbot cost approximately $0.0057, compared to an estimated €2.72 for equivalent human tutoring.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
What is the application?
Who is the user?
Who age?
Study design
