Date
AI Overview
Why it matters
Direct language model prompting to diagnose student understanding in conversational tutoring lacks a psychological foundation and reliable outputs. Tracking evolving cognitive states across multi-turn interactions remains difficult.
Study at a glance
- Question: Can a multi-agent framework improve student cognitive diagnosis and tutoring efficiency in conversational learning dialogues?
- Setting and data: 2,861 simulated tutoring dialogues from MathDial and 116 authentic Khan Academy tutoring conversations from CoMTA.
- Method: Multi-agent framework development, benchmark evaluations, ablation experiments, and tutoring simulations.
- AI tools: ParLD, GPT-4.1, GPT-4o, InstructGPT
Key findings
- ParLD with GPT-4.1 achieved 68.72% accuracy on MathDial and 57.26% on CoMTA, outperforming traditional knowledge tracing models.
- Ablation tests revealed that removing the Behavior Previewer caused the largest decline in student learning outcome prediction accuracy.
- In tutoring simulations, ParLD-guided instruction helped 72.22% of simulated students solve math problems correctly, compared to 56.48% with direct responses.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
Learning diagnosis is a critical task that monitors students' cognitive state during educational activities, with the goal of enhancing learning outcomes. With advancements in language models (LMs), many AI-driven educational studies have shifted towards conversational learning scenarios, where students engage in multi-turn interactive dialogues with tutors. However, conversational learning diagnosis remains underdeveloped, and most existing techniques acquire students' cognitive state through intuitive instructional prompts on LMs to analyze the dialogue text. This direct prompting approach lacks a solid psychological foundation and fails to ensure the reliability of the generated analytical text. In this study, we introduce ParLD, a preview-analyze-reason framework for conversational learning diagnosis, which leverages multi-agent collaboration to diagnose students' cognitive state over multiple dialogue turns. Specifically, ParLD comprises three main components: (1) Behavior Previewer, which generates a student behavior schema based on previous states and learning content; (2) State Analyzer, which diagnoses the tutor-student dialogue and behavior schema to update the cognitive state; and (3) Performance Reasoner, which predicts the student's future responses and provides verifiable feedback to support ParLD's self-reflection with the Chain Reflector. They operate sequentially and iteratively during each interaction turn to diagnose the student's cognitive state. We conduct experiments to evaluate both performance prediction and tutoring support, emphasizing the effectiveness of ParLD in providing reliable and insightful learning diagnosis.
Research Categories
What is the application?
Why use AI?
Study design
Publication Source
Publisher
arXiv
