Date
Publisher
arXiv
Large language models (LLMs) are increasingly positioned as solutions for
education, yet evaluations often reduce their impact to narrow performance
metrics. This paper reframes the question by asking "what kind of impact should
LLMs have in education?" Drawing on Biesta's tripartite account of good
education: qualification, socialisation, and subjectification, we present a
meta-analysis of 133 experimental and quasi-experimental studies (k = 188).
Overall, the impact of LLMs on student learning is positive but uneven. Strong
effects emerge in qualification, particularly when LLMs function as tutors in
sustained interventions. Socialisation outcomes appear more variable,
concentrated in sustained, reflective interventions. Subjectification, linked
to autonomy and learner development, remains fragile, with improvements
confined to small-scale, long-term studies. This purpose-level view highlights
design as the decisive factor: without scaffolds for participation and agency,
LLMs privilege what is easiest to measure while neglecting broader aims of
education. For HCI and education, the issue is not just whether LLMs work, but
what futures they enable or foreclose.
Study at a glance
- Question: How do large language models affect students' knowledge acquisition, socialization, and personal autonomy?
- Setting and data: 133 experimental and quasi-experimental studies (188 effect sizes) covering kindergarten through higher education across global regions, published from November 2022 to March 2025.
- Method: Systematic review and meta-analysis using random-effects models and moderator analyses.
Why it matters
Evaluations often reduce the impact of large language models to narrow test scores, leaving little evidence on whether they support student collaboration, social participation, and learner agency.
Key findings
- Large language models showed significant positive overall effects on qualification (g = 0.751), socialization (g = 0.745), and learner subjectification (g = 0.654), though subjectification was the least stable.
- Qualification gains were largest when models functioned as tutors (g = 0.902) or partners (g = 0.842) rather than tools, and in interventions lasting over eight weeks (g = 1.024).
- Socialization gains were strongest in interventions lasting over eight weeks and using personalized learning, while subjectification effects diminished sharply in cohorts larger than 100 students.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
What is the application?
Who is the user?
