Date
Publisher
arXiv
Problem solving plays an essential role in science education, and generative AI (GAI) chatbots have emerged as a promising tool for supporting students' science problem solving. However, general-purpose chatbots (e.g., ChatGPT), which often provide direct, ready-made answers, may lead to students' cognitive offloading. Prior research has rarely focused on custom chatbots for facilitating students' science problem solving, nor has it examined how they differently influence problem-solving processes and performance compared to general-purpose chatbots. To address this gap, we developed a pedagogy-informed custom GAI chatbot grounded in the Socratic questioning method, which supports students by prompting them with guiding questions. This study employed a within-subjects counterbalanced design in which 48 secondary school students used both custom and general-purpose chatbot to complete two science problem-solving tasks. 3297 student-chatbot dialogues were collected and analyzed using Heterogeneous Interaction Network Analysis (HINA). The results showed that: (1) students demonstrated significantly higher interaction intensity and cognitive interaction diversity when using custom chatbot than using general-purpose chatbot; (2) students were more likely to follow custom chatbot's guidance to think and reflect, whereas they tended to request general-purpose chatbot to execute specific commands; and (3) no statistically significant difference was observed in students' problem-solving performance evaluated by solution quality between two chatbot conditions. This study provides novel theoretical insights and empirical evidence that custom chatbots are less likely to induce cognitive offloading and instead foster greater cognitive engagement compared to general-purpose chatbots. This study also offers insights into the design and integration of GAI chatbots in science education.
Study at a glance
- Question: How do Socratic custom and general-purpose chatbots differ in shaping students' science problem-solving processes and performance?
- Setting and data: Hong Kong, 48 Form 1 secondary school girls aged 11–14 across two classes, analyzing 3,297 dialogue turns and submitted solutions.
- Method: Within-subjects counterbalanced experiment with content coding, network analysis (Heterogeneous Interaction Network Analysis), and linear mixed-effects modeling.
Why it matters
General-purpose generative AI chatbots risk cognitive offloading by giving ready-made answers. Little is known about how custom, pedagogy-informed chatbots compare in supporting students' science problem-solving processes and learning.
Key findings
- Students showed significantly higher interaction intensity and greater cognitive interaction diversity when using the custom Socratic chatbot compared to the general-purpose chatbot.
- Students predominantly followed guided problem-solving steps and reflected on solutions with the custom chatbot, whereas they frequently requested direct answers, formatting, or translation with the general-purpose chatbot.
- No statistically significant difference was found in students' problem-solving performance or solution quality scores between the custom and general-purpose chatbot conditions.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
What is the application?
Who is the user?
Who age?
Why use AI?
Study design
