Search and Filter

Can AI deliver appropriate support for diverse student profiles?: A large-scale evaluation

Date

AI Overview

Why it matters

Large language models could potentially support students through learning analytics-informed advising, but it is unclear whether they can adaptively recommend appropriate support across diverse student needs, characteristics, and contexts.

Study at a glance
  • Question: Can large language models reliably and adaptively recommend appropriate support for diverse student profiles?
  • Setting and data: 4,500 synthetic student vignettes generated by three large language models.
  • Method: Model evaluation using correlation and variance analyses on AI-generated support recommendations across synthetic student profiles.
  • AI tools: GPT-5-mini, Mistral-Medium-2508, Qwen-Plus
Key findings
  • The evaluated language models showed limited sensitivity to student characteristics and learning analytics indicators when generating support recommendations.
  • Support recommendations differed considerably and inconsistently across the three tested models.
  • Current models are not yet reliable as prescriptive tools for delivering ethical, consistent, and reliable student support at scale.

Summary drafted with AI from the paper’s abstract and may contain errors. See the full paper for details.

Large language models (LLMs) have the potential to offer a novel approach to student support through learning analytics (LA)–informed advising. This study examines such potential by exploring whether current LLMs can adaptively recommend appropriate support across student needs, characteristics, and contexts. To do that, we generated 4500 synthetic student vignettes using three LLMs (GPT-5-mini, Mistral-Medium-2508, and Qwen-Plus). Each vignette included a single behavioral student trait, classified as either high- or low-performing, and one positive or negative LA indicator describing student learning behavior. These were randomly selected from pre-defined literature-informed lists to provide the LLM with contextual information about the student. The LLMs then generated recommendations for support, including the level of support needed and timing. We examined whether the three LLMs behaved consistently with each other and whether they responded differentially to LA indicators and other learner and context characteristics. For this purpose, we analysed the generated data using Pearson correlations with False Discovery Rate correction and analyses of variance conducted separately for each LLM. Findings revealed limited sensitivity to student characteristics as indicated by LA indicators and the recommended support. In addition, we detected considerable inconsistencies in LLM-generated support recommendations across tested LLMs. These results suggest that current LLMs are not yet reliable as prescriptive models for providing student support at scale in an ethical, consistent, and reliable manner.