Date
Publisher
arXiv
How can we test whether state-of-the-art generative models, such as Blender
and GPT-3, are good AI teachers, capable of replying to a student in an
educational dialogue? Designing an AI teacher test is challenging: although
evaluation methods are much-needed, there is no off-the-shelf solution to
measuring pedagogical ability. This paper reports on a first attempt at an AI
teacher test. We built a solution around the insight that you can run
conversational agents in parallel to human teachers in real-world dialogues,
simulate how different agents would respond to a student, and compare these
counterpart responses in terms of three abilities: speak like a teacher,
understand a student, help a student. Our method builds on the reliability of
comparative judgments in education and uses a probabilistic model and Bayesian
sampling to infer estimates of pedagogical ability. We find that, even though
conversational agents (Blender in particular) perform well on conversational
uptake, they are quantifiably worse than real teachers on several pedagogical
dimensions, especially with regard to helpfulness (Blender: {\Delta} ability =
-0.75; GPT-3: {\Delta} ability = -0.93).
Study at a glance
- Question: How can we test whether conversational AI models are good teachers capable of replying effectively to students?
- Setting and data: Real-world dialogues between human teachers and students.
- Method: Compares simulated AI responses to human teacher responses in student dialogues using comparative judgments (probabilistic model and Bayesian sampling).
Why it matters
Although evaluation methods are much-needed to test whether generative models are good AI teachers in educational dialogues, there is no off-the-shelf solution to measuring pedagogical ability.
Key findings
- In real-world educational dialogues, conversational AI models performed worse than human teachers across several pedagogical dimensions.
- Both tested AI models were notably less helpful to students than human teachers were.
- Conversational AI models, particularly Blender, performed well at conversational uptake when responding to students.
Summary drafted with AI from the paper’s abstract and may contain errors. See the full paper for details.
What is the application?
Who is the user?
Who age?
Why use AI?
Study design
