Date
Publisher
arXiv
While Large Language Models (LLMs) can provide personalized support in learning, several studies have raised concerns regarding their use in education. Importantly, learning depends on how students engage with LLMs. This study examined how two types of LLM-based tutors shape students' prompting practices, learning, and subsequent LLM-use: a Socratic-Guidance (SG) tutor, which structures interaction through dialogic questioning, and a Prompt-Refinement (PR) tutor that guides the formulation of effective prompts. We conducted a two-phase study in a graduate-level mobile robotics course: 66 students used either the SG or PR tutor during a 6-week intervention, followed by 52 students using an unconstrained LLM during a 3-week course project. Results show that while the SG- and PR tutors led to similar task performance and prompting patterns during guided use, they differ in learning outcomes and later LLM-use. SG-students, relative to PR-student, achieved higher learning gains in later sessions, and were more likely to adopt understanding-driven prompting strategies, which are predictive of higher understanding, when using an unconstrained LLM. Although learners perceived the SG tutor as less efficient, the findings suggest that Socratic guidance supports the development of students' capacity to learn with LLMs over time, highlighting its importance for LLM tutor design.
Study at a glance
- Question: How do Socratic questioning and prompt-refinement AI tutors affect student learning, performance, and subsequent independent model use?
- Setting and data: Graduate mobile robotics course at EPFL, Switzerland; 66 students in lab sessions and 52 students in a project phase.
- Method: Randomized controlled trial (mixed-methods, between-subjects experiment).
Why it matters
Students using AI models risk cognitive debt and reduced engagement. Structuring interactions through pedagogical designs may help students develop productive, independent ways of learning with these systems beyond instructional settings.
Key findings
- Students using the Socratic tutor achieved significantly greater learning gains by the final lab session than those using the prompt-refinement tutor, despite having lower pre-test scores.
- During an unconstrained project phase, students previously assigned to the Socratic tutor were significantly more likely to adopt understanding-driven prompting patterns that predicted higher project understanding.
- Students perceived the prompt-refinement tutor as significantly more effective for learning and writing useful prompts, even though it produced lower learning gains.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
What is the application?
Who is the user?
Who age?
Why use AI?
Study design
