Date
Publisher
arXiv
Large language models (LLMs) increasingly serve as educational tools, yet evaluating their teaching capabilities remains challenging due to the resource-intensive, context-dependent, and methodologically complex nature of teacher-student interactions. We introduce EducationQ, a multi-agent dialogue framework that efficiently assesses teaching capabilities through simulated dynamic educational scenarios, featuring specialized agents for teaching, learning, and evaluation. Testing 14 LLMs across major AI Organizations (OpenAI, Meta, Google, Anthropic, and others) on 1,498 questions spanning 13 disciplines and 10 difficulty levels reveals that teaching effectiveness does not correlate linearly with model scale or general reasoning capabilities - with some smaller open-source models outperforming larger commercial counterparts in teaching contexts. This finding highlights a critical gap in current evaluations that prioritize knowledge recall over interactive pedagogy. Our mixed-methods evaluation, combining quantitative metrics with qualitative analysis and expert case studies, identifies distinct pedagogical strengths employed by top-performing models (e.g., sophisticated questioning strategies, adaptive feedback mechanisms). Human expert evaluations show 78% agreement with our automated qualitative analysis of effective teaching behaviors, validating our methodology. EducationQ demonstrates that LLMs-as-teachers require specialized optimization beyond simple scaling, suggesting next-generation educational AI prioritize targeted enhancement of specific pedagogical effectiveness.
Study at a glance
- Question: How effectively can large language models teach simulated students across diverse subjects in multi-turn interactive dialogues?
- Setting and data: 1,498 undergraduate- to PhD-level questions from GPQA Diamond and MMLU-Pro across 13 academic disciplines; 14 language models evaluated as teachers.
- Method: Multi-agent dialogue simulation evaluating learning gains from pre- to post-test, combined with automated qualitative analysis and human expert review (mixed-methods).
Why it matters
Current benchmarks test whether language models recall knowledge rather than their ability to teach through interactive dialogue, formative assessment, and adaptive feedback.
Key findings
- General knowledge benchmarks do not predict teaching effectiveness, as some smaller open-source models outperformed larger commercial counterparts.
- Llama 3.1 70B Instruct achieved the highest overall student learning gain at 11.01% across 13 disciplines, followed by Gemini 1.5 Pro 002 at 7.48%.
- Human education experts agreed with the automated qualitative evaluation in 78% of reviewed dialogue comparisons, confirming distinct teaching strategies drove learning gains.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
What is the application?
Who age?
Why use AI?
Study design
