Date
AI Overview
Why it matters
Evaluations of educational large language models often focus on technical performance or learning outcomes while overlooking multi-turn dialogue dynamics. Analyzing dialogue act sequences helps uncover the mechanisms and pedagogical strategies that foster student learning.
Study at a glance
- Question: What sequential dialogue patterns appear in effective versus less effective learner-chatbot tutoring sessions?
- Setting and data: 6 English language learners (3 high progress, 3 low progress), 12 total practice sessions, 1,020 manually annotated dialogue turns.
- Method: Dialogue act annotation, sequential pattern mining (CM-SPAM algorithm), and qualitative transcript analysis.
Key findings
- High-progress students produced longer utterances during speaking practice, averaging 15.3 words per turn compared to 11.5 words for low-progress students.
- High-progress sessions exhibited a bot question–student response–student question sequence, whereas low-progress sessions uniquely featured consecutive bot and student questions without a student response.
- Transcript analysis suggests that students in the low-progress group asked questions instead of answering because they had difficulty understanding the chatbot's initial question.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
Dialogue plays a crucial role in educational settings, yet existing
evaluation methods for educational applications of large language models (LLMs)
primarily focus on technical performance or learning outcomes, often neglecting
attention to learner-LLM interactions. To narrow this gap, this AIED Doctoral
Consortium paper presents an ongoing study employing a dialogue analysis
approach to identify effective pedagogical strategies from learner-LLM
dialogues. The proposed approach involves dialogue data collection, dialogue
act (DA) annotation, DA pattern mining, and predictive model building. Early
insights are outlined as an initial step toward future research. The work
underscores the need to evaluate LLM-based educational applications by focusing
on dialogue dynamics and pedagogical strategies.
Research Categories
What is the application?
Who is the user?
Why use AI?
Study design
Publication Source
Publisher
arXiv
