Date
AI Overview
Why it matters
Educational chatbots powered by large language models can scaffold student learning, but their effectiveness depends on student prompts, which often take the form of questions that are difficult for students to articulate.
Study at a glance
- Question: What types of questions do STEM students ask educational chatbots across formative self-study and summative coursework tasks?
- Setting and data: UK and US universities, 244 undergraduate engineering and computer science students, 6,113 chatbot messages across two courses.
- Method: Automated text classification using 11 large language models and three human raters applying four question schemas.
- AI tools: gemini-2.0-flash, gpt-4o-mini, gpt-5.1, gpt-4.1, deepseek-r1:70b, llama4:128x17b
Key findings
- Procedural questions asking how to proceed or what steps to use predominated in both learning contexts across all four classification schemas.
- Students asked procedural questions more frequently during summative coursework assignments than during formative self-study tasks across all schemas.
- Eleven large language model raters achieved moderate-to-good agreement when classifying student questions, demonstrating higher consistency than the three human raters.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
Providing scaffolding through educational chatbots built on Large Language Models (LLM) has potential risks and benefits that remain an open area of research. When students navigate impasses, they ask for help by formulating impasse-driven questions. Within interactions with LLM chatbots, such questions shape the user prompts and drive the pedagogical effectiveness of the chatbot's response. This paper focuses on such student questions from two datasets of distinct learning contexts: formative self-study, and summative assessed coursework. We analysed 6,113 messages from both learning contexts, using 11 different LLMs and three human raters to classify student questions using four existing schemas. On the feasibility of using LLMs as raters, results showed moderate-to-good inter-rater reliability, with higher consistency than human raters. The data showed that 'procedural' questions predominated in both learning contexts, but more so when students prepare for summative assessment. These results provide a basis on which to use LLMs for classification of student questions. However, we identify clear limitations in both the ability to classify with schemas and the value of doing so: schemas are limited and thus struggle to accommodate the semantic richness of composite prompts, offering only partial understanding the wider risks and benefits of chatbot integration. In the future, we recommend an analysis approach that captures the nuanced, multi-turn nature of conversation, for example, by applying methods from conversation analysis in discursive psychology.
Research Categories
What is the application?
Who is the user?
Who age?
Why use AI?
Study design
AI tools
Datasets
Publication Source
Publisher
arXiv
