Date
Publisher
arXiv
This study explores the use of generative AI for automating the
classification of tutors' Dialogue Acts (DAs), aiming to reduce the time and
effort required by traditional manual coding. This case study uses the
open-source CIMA corpus, in which tutors' responses are pre-annotated into four
DA categories. Both GPT-3.5-turbo and GPT-4 models were tested using tailored
prompts. Results show that GPT-4 achieved 80% accuracy, a weighted F1-score of
0.81, and a Cohen's Kappa of 0.74, surpassing baseline performance and
indicating substantial agreement with human annotations. These findings suggest
that generative AI has strong potential to provide an efficient and accessible
approach to DA classification, with meaningful implications for educational
dialogue analysis. The study also highlights the importance of task-specific
label definitions and contextual information in enhancing the quality of
automated annotation. Finally, it underscores the ethical considerations
associated with the use of generative AI and the need for responsible and
transparent research practices. The script of this research is publicly
available at
https://github.com/liqunhe27/Generative-AI-for-educational-dialogue-act-tagging.
Study at a glance
- Question: How accurately can generative AI categorize tutors' dialogue compared to human coding?
- Setting and data: Open-source CIMA corpus containing tutors' responses pre-annotated into four dialogue act categories.
- Method: Case study testing two AI models with tailored prompts against human-annotated tutoring dialogues.
Why it matters
Traditional manual coding of tutors' dialogue acts requires time and effort. Automating this classification with generative AI aims to reduce that burden.
Key findings
- In an open-source tutoring dialogue dataset, generative AI achieved substantial agreement with human annotators when categorizing tutor responses into four dialogue types.
- Providing task-specific definitions and conversational context enhanced the quality of the automated dialogue classifications.
- Generative AI demonstrated strong potential to serve as an efficient and accessible tool for educational dialogue analysis.
Summary drafted with AI from the paper’s abstract and may contain errors. See the full paper for details.
What is the application?
Who age?
Why use AI?
Study design
