The Impact of Large Language Models on Students: A Randomised Study of Socratic vs. Non-Socratic AI and the Role of Step-by-Step Reasoning

Purpose How does integrating Large Language Models (LLMs) into classroom activities influence students' learning? Using a randomised controlled experiment, we examined whether LLMs' step-by-step explanatory capabilities improve students' performance on complex tasks. Additionally, we compare the effects of Socratic AI-LLMpowered chatbots designed to encourage critical thinking through incremental guidance with Non-Socratic AI in shaping students' engagement, interactions, and learning outcomes. Methods The study was randomised with 122 students aged 14 to 18 from two high schools. Participants engaged in problem-solving and writing tasks under different experimental conditions. The impact of step-by-step reasoning was tested on a simple prediction task (guessing the value of coins in a jar). Conversely, the effect of Socratic AI on students was assessed through objective test scores on various tasks, while attitudes were measured using self-reported surveys. Results While all students received AI-generated predictions, those exposed to step-by-step reasoning demonstrated significantly higher accuracy in the prediction task, highlighting the benefits of AI reasoning. Socratic AI fostered significantly greater engagement and interaction_ however, it did not yield significant improvements in learning, and a higher fraction of students perceived it as less helpful. Furthermore, despite students generally perceiving AI assistance as beneficial, they exhibited limited retention, struggling to apply learned concepts to new situations when we removed AI assistance. Conclusion These findings contribute to the ongoing debate on integrating LLM-powered chatbots in education. They emphasize the importance of fostering effective student-AI interactions and incorporating pedagogically grounded principles into AI design to enhance educational outcomes.