Who Benefits from LLM-Generated Learning Support Messages? Causal Evidence on Treatment Effect Heterogeneity

This study examines the impact of LLM-generated learning support messages on learning outcomes in a large-scale open and distance learning course (). Effects are estimated for the full randomized sample, among compliers, and across heterogeneous learner profiles. Assignment to the experimental group led to small but statistically significant improvements in course scores (0.60 points, ) and final exam participation (1.9 percentage points). Effects were larger among learners who engaged with the messages ( points on course scores). However, only 29% of treated learners interacted with the messages, indicating low compliance. Causal machine-learning methods reveal substantial heterogeneity in responsiveness. Model-based estimates suggest positive effects for approximately two-thirds of learners, particularly among those with moderate prior academic performance and among older age cohorts, with effects peaking around ages 34–40. No statistically significant effects are observed for final exam scores or for the measured engagement outcomes. The results indicate that the intervention affects learners differently and that targeted deployment based on learner profiles may be more effective than uniform exposure.