Date
Publisher
arXiv
While predictive models are increasingly common in higher education, causal evidence regarding the interventions they trigger remains rare. This study evaluates an AI-guided student support system at a large university using doubly robust propensity score matching. We advance the methodology for learning analytics evaluation by leveraging time-aligned, dynamic AI probability of success scores to match 1,859 treated students to controls, thereby mitigating the selection and immortal time biases often overlooked in observational studies. Results indicate that the intervention effectively stabilised precarious trajectories, and compared to the control group, supported students significantly reduced their course failure rates and achieved higher cumulative grades. However, effects on the speed of qualification completion were positive but statistically constrained. We interpreted these findings through Activity Theory, framing the intervention as a socio-technical brake that interrupts and slows the accumulation of academic failure among at-risk students. The student support-AI configuration successfully resolved the primary contradiction of immediate academic risk, but secondary contradictions within institutional structures limited the acceleration of degree completion. We conclude that while AI-enabled support effectively arrests decline, translating this stability into faster progression requires aligning intervention strategies with broader institutional governance.
Study at a glance
- Question: How does proactive AI-guided student support affect academic performance and progression for at-risk undergraduate students?
- Setting and data: Massey University in New Zealand, 2022 to 2024, 1,859 supported at-risk undergraduate students and matched controls.
- Method: Propensity score matching with doubly robust estimation.
Why it matters
Predictive models are common in higher education, but causal evidence regarding the interventions they trigger remains rare, and evaluations often lack a theoretical grounding of how support is embedded in institutional systems.
Key findings
- Supported at-risk students failed 0.78 fewer courses on average in the intervention year compared to matched peers, equating to 78 avoided course failures per 100 students.
- Supported students achieved cumulative mean grades 8.82 points higher and cumulative pass rates roughly 14 percentage points higher than comparable peers.
- The intervention was associated with an estimated 7.94 percentage point increase in qualification completion progress, though the effect was statistically imprecise.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
What is the application?
Who is the user?
Who age?
Why use AI?
Study design
