Date
Publisher
arXiv
Large language models (LLMs) are increasingly positioned as solutions for
education, yet evaluations often reduce their impact to narrow performance
metrics. This paper reframes the question by asking "what kind of impact should
LLMs have in education?" Drawing on Biesta's tripartite account of good
education: qualification, socialisation, and subjectification, we present a
meta-analysis of 133 experimental and quasi-experimental studies (k = 188).
Overall, the impact of LLMs on student learning is positive but uneven. Strong
effects emerge in qualification, particularly when LLMs function as tutors in
sustained interventions. Socialisation outcomes appear more variable,
concentrated in sustained, reflective interventions. Subjectification, linked
to autonomy and learner development, remains fragile, with improvements
confined to small-scale, long-term studies. This purpose-level view highlights
design as the decisive factor: without scaffolds for participation and agency,
LLMs privilege what is easiest to measure while neglecting broader aims of
education. For HCI and education, the issue is not just whether LLMs work, but
what futures they enable or foreclose.
Study at a glance
- Question: How do large language models affect students across academic qualification, socialisation, and subjectification?
- Setting and data: 133 experimental and quasi-experimental studies (188 effect sizes) published globally between November 2022 and March 2025, largely in Asian higher education.
- Method: Systematic literature review and random-effects meta-analysis.
Why it matters
Evaluations of large language models often focus narrowly on test scores. This measurement-first focus risks neglecting essential educational purposes, including social participation and the development of student agency and autonomy.
Key findings
- Large language models showed significant positive overall effects on academic qualification (g = 0.751), socialisation (g = 0.745), and learner subjectification (g = 0.654).
- Gains in qualification were largest when models served as interactive tutors (g = 0.902) rather than tools (g = 0.436), and in interventions lasting over eight weeks.
- Subjectification effects were strongest in programs longer than eight weeks and in cohorts with 50 or fewer students, diminishing sharply in groups larger than 100.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
What is the application?
Who is the user?
Study design
