Can AI deliver appropriate support for diverse student profiles?: A large-scale evaluation

Large language models (LLMs) have the potential to offer a novel approach to student support through learning analytics (LA)–informed advising. This study examines such potential by exploring whether current LLMs can adaptively recommend appropriate support across student needs, characteristics, and contexts. To do that, we generated 4500 synthetic student vignettes using three LLMs (GPT-5-mini, Mistral-Medium-2508, and Qwen-Plus). Each vignette included a single behavioral student trait, classified as either high- or low-performing, and one positive or negative LA indicator describing student learning behavior. These were randomly selected from pre-defined literature-informed lists to provide the LLM with contextual information about the student. The LLMs then generated recommendations for support, including the level of support needed and timing. We examined whether the three LLMs behaved consistently with each other and whether they responded differentially to LA indicators and other learner and context characteristics. For this purpose, we analysed the generated data using Pearson correlations with False Discovery Rate correction and analyses of variance conducted separately for each LLM. Findings revealed limited sensitivity to student characteristics as indicated by LA indicators and the recommended support. In addition, we detected considerable inconsistencies in LLM-generated support recommendations across tested LLMs. These results suggest that current LLMs are not yet reliable as prescriptive models for providing student support at scale in an ethical, consistent, and reliable manner.