Date
Publisher
arXiv
This paper describes the results of the first shared task on the generation
of teacher responses in educational dialogues. The goal of the task was to
benchmark the ability of generative language models to act as AI teachers,
replying to a student in a teacher-student dialogue. Eight teams participated
in the competition hosted on CodaLab. They experimented with a wide variety of
state-of-the-art models, including Alpaca, Bloom, DialoGPT, DistilGPT-2,
Flan-T5, GPT-2, GPT-3, GPT- 4, LLaMA, OPT-2.7B, and T5-base. Their submissions
were automatically scored using BERTScore and DialogRPT metrics, and the top
three among them were further manually evaluated in terms of pedagogical
ability based on Tack and Piech (2022). The NAISTeacher system, which ranked
first in both automated and human evaluation, generated responses with GPT-3.5
using an ensemble of prompts and a DialogRPT-based ranking of responses for
given dialogue contexts. Despite the promising achievements of the
participating teams, the results also highlight the need for evaluation metrics
better suited to educational contexts.
Study at a glance
- Question: How well can generative AI models act as teachers replying to students in educational conversations?
- Setting and data: Not stated in the abstract
- Method: Evaluation of multiple AI models using automated metrics and human ratings of pedagogical ability (shared task)
Why it matters
Shared benchmarks are needed to evaluate the ability of generative language models to act as AI teachers replying to students in educational dialogues.
Key findings
- In educational dialogues, the top AI teacher system achieved the highest ratings in both automated scoring and human pedagogical evaluations.
- Generative language models demonstrated promising capabilities when generating conversational responses to students as AI teachers.
- Existing automated metrics were found to be insufficiently suited for evaluating AI teacher responses in educational settings.
Summary drafted with AI from the paper’s abstract and may contain errors. See the full paper for details.
What is the application?
Who is the user?
Why use AI?
Study design
