Date
Publisher
arXiv
Artificial intelligence (AI) systems in education are developing on timescales that sit uneasily with conventional evaluation. By the time a large-scale trial has been designed, delivered, analysed and published, the technology under study may have changed materially. This creates a temporal problem for evidence-informed education: the need for timely evidence can encourage reliance on weak observational or usage data, while conventional rigorous evaluation may produce evidence too slowly to guide rapidly evolving practice. We examine teacher-led micro-randomised controlled trials (micro-RCTs) as one response to this problem. The empirical case is a four-week multisite individually randomised evaluation of Medly, an AI-powered tutoring platform, in GCSE Biology, Chemistry and Physics in English secondary schools. Of 929 students completing baseline assessment, 644 completed post-testing. In the primary ITT analysis, students allocated to Medly achieved higher post-test attainment than students undertaking business-as-usual self-directed revision (Hedges' g = 0.33, 95% CI 0.18 to 0.48). Positive estimates were observed in Physics (g = 0.31), Chemistry (g = 0.32) and Biology (g = 0.52), with no evidence of differential impact by disadvantage status. Greater platform engagement was associated with higher attainment, but these post-randomisation analyses are treated as exploratory rather than causal. Attrition was substantial (30.7%), outcome measures were curriculum-aligned rather than standardised, and process evaluation response was limited. We therefore interpret the findings as preliminary. We argue that the value of micro-RCTs for educational AI lies not in replacing definitive evaluation with small studies, but in enabling a rapid, cumulative evaluation architecture in which randomised estimates can be generated, replicated and updated as technologies and their implementation evolve.
Study at a glance
- Question: Does using an AI tutoring platform improve secondary school students' science exam revision scores compared to usual revision?
- Setting and data: Year 9 and 10 students across English mainstream secondary schools, 929 baseline students and 644 post-test completers across 39 teacher trials.
- Method: Multisite individually randomized controlled trial (practitioner-led micro-RCTs) with mixed-effects models.
Why it matters
Educational AI evolves faster than conventional evaluations can report, creating a temporal lag where interventions change before findings appear. Rigorous rapid evidence is needed so schools avoid relying solely on weak observational data.
Key findings
- Students allocated to the AI tutoring platform scored significantly higher on science post-tests than those doing business-as-usual revision, with an effect size of 0.33.
- Positive test gains appeared across all subjects, with effect sizes of 0.31 in Physics, 0.32 in Chemistry, and 0.52 in Biology.
- There was no clear evidence that the tutoring platform's impact differed by students' Pupil Premium disadvantage status.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
What is the application?
Who age?
Why use AI?
Study design
