Date
AI Overview
Why it matters
Understanding whether personalized recommendations increase engagement in educational technology is key to determining how algorithmic interventions perform at scale compared to early experimental evaluations.
Study at a glance
- Question: Can personalized algorithmic recommendations improve student engagement on an educational technology platform?
- Setting and data: Freadom English-learning app for children in India; 7,750 students in an RCT, followed by five months of platform deployment.
- Method: Randomized controlled trial followed by a regression discontinuity design.
- AI tools: Freadom
Key findings
- Personalized recommendations in a single content section increased student engagement by 60% in that section and 14% across the entire app.
- During scaled deployment, regression discontinuity estimates of user engagement exceeded the initial experimental benchmark by a factor of 2.5.
- Personalization gains concentrated in users' first weeks and diminished thereafter as predicted match quality declined and content availability became the binding constraint.
Summary drafted with AI from the paper’s abstract and may contain errors. See the full paper for details.
Can personalized recommendations improve engagement in educational technology? We design, test, and scale a collaborative filtering system for Freadom, an English-learning app for Indian children. A randomized controlled trial (RCT) with 7,750 students shows that personalization, deployed in a single content section, increases engagement by 60% in that section and by 14% app-wide. We then exploit an eligibility threshold in a regression discontinuity design (RDD) to track effects over five months of deployment. For user cohorts receiving personalization during deployment, RDD estimates exceed RCT benchmark by a factor of 2.5, opposite of the “voltage drop" typically observed in policy scale-ups. This provides evidence that, for algorithmic interventions, RCT estimates may be lower bounds on scaled impact rather than upper bounds. However, personalization benefits are front-loaded. Gains concentrate in users’ first weeks, with diminishing returns thereafter. This pattern, combined with the sharp decline in predicted match quality as users exhaust their best content matches, suggests that content availability rather than algorithmic sophistication becomes the binding constraint.
Research Categories
What is the application?
Who is the user?
Who age?
Why use AI?
AI tools
Publication Source
Publisher
NBER
