Date
Publisher
arXiv
While Large Language Models (LLMs) can provide personalized support in learning, several studies have raised concerns regarding their use in education. Importantly, learning depends on how students engage with LLMs. This study examined how two types of LLM-based tutors shape students' prompting practices, learning, and subsequent LLM-use: a Socratic-Guidance (SG) tutor, which structures interaction through dialogic questioning, and a Prompt-Refinement (PR) tutor that guides the formulation of effective prompts. We conducted a two-phase study in a graduate-level mobile robotics course: 66 students used either the SG or PR tutor during a 6-week intervention, followed by 52 students using an unconstrained LLM during a 3-week course project. Results show that while the SG- and PR tutors led to similar task performance and prompting patterns during guided use, they differ in learning outcomes and later LLM-use. SG-students, relative to PR-student, achieved higher learning gains in later sessions, and were more likely to adopt understanding-driven prompting strategies, which are predictive of higher understanding, when using an unconstrained LLM. Although learners perceived the SG tutor as less efficient, the findings suggest that Socratic guidance supports the development of students' capacity to learn with LLMs over time, highlighting its importance for LLM tutor design.
Study at a glance
- Question: How do Socratic guidance and prompt refinement AI tutors affect students' learning, prompting, and later independent AI use?
- Setting and data: A graduate-level mobile robotics course at EPFL, Switzerland; 66 students in lab sessions and 52 students in a project phase.
- Method: Randomized controlled trial with repeated measures and observational follow-up (mixed methods, between-subjects design).
Why it matters
Students' use of AI tools risks reducing active cognitive engagement. To support productive, long-term learning with language models, educators need to know whether to structure AI responses or guide how students formulate prompts.
Key findings
- Students using the Socratic tutor achieved significantly greater learning gains by the final lab session than those using the prompt-refinement tutor, despite lower initial task performance.
- During an unconstrained course project, students previously assigned to the Socratic tutor were significantly more likely to adopt understanding-driven prompting strategies.
- Students rated the prompt-refinement tutor as significantly more effective for learning and writing useful prompts, even though it led to lower learning gains.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
What is the application?
Who is the user?
Who age?
Why use AI?
Study design
