Date
Publisher
arXiv
Large language model (LLM) tutors are being deployed in introductory physics courses at a scale that produces transcript corpora far larger than traditional qualitative coding can absorb. A central question for physics education research (PER) is empirical and prior to any claim about effectiveness: what do students actually say to these tutors? We address this question for one Socratic AI tutor deployed in an introductory calculus-based mechanics course by building a bottom-up taxonomy of student discourse. Each student turn is assigned an emergent free-text label by an LLM coder using the surrounding conversational context; near-paraphrase labels are then consolidated into a smaller set of discourse categories using a similarity-based grouping procedure. The procedure is validated against a stratified human-coded sample. The resulting taxonomy of 357 categories is strikingly concentrated: the top 25 categories cover roughly half of all student turns, and two thematic bands: equation-handling and meta-procedural requests together dominate the head of the distribution. The substantive contribution is the taxonomy itself: a description of the discourse PER researchers can expect to encounter when students work with an AI tutor of this design, including a striking prevalence of meta-procedural turns in which students cede strategic control to the tutor
Why it matters
Large language model tutors produce transcripts far larger than traditional qualitative coding can absorb. Prior to making claims about effectiveness, researchers must first answer a central, empirical question: what do students actually say to these tutors?
Key findings
- In an introductory calculus-based mechanics course, just 25 categories accounted for roughly half of all student messages to a Socratic AI tutor.
- Student dialogue was dominated by two primary topics: handling equations and making procedural requests about the problem-solving process.
- Students strikingly often ceded strategic control of their work to the AI tutor through meta-procedural requests.
Summary drafted with AI from the paper’s abstract and may contain errors. See the full paper for details.
What is the application?
Who is the user?
Who age?
Why use AI?
Study design
