Date
Publisher
arXiv
This study investigated effective strategies for developing a custom GPT to code classroom dialogue. While classroom dialogue is widely recognised as a crucial element of education, its analysis remains challenging due to the need for a nuanced understanding of dialogic functions and the labour-intensive nature of manual transcript coding. Recent advancements in large language models (LLMs) offer promising avenues for automating this process. However, existing studies predominantly focus on training large-scale models or evaluating pre-trained models with fixed codebooks, the outcomes of which are often not applicable, or the methods are not replicable for dialogue researchers working with small datasets or employing customised coding schemes. Using MyGPT - a GPT-4-based customised GPT system configured for dialogue analysis - as a case, this study evaluates its baseline performance in coding classroom dialogue with a human codebook and examines how performance varies with different example inputs under a controlled variable design. Through a design-based research approach, this study explores a set of practical strategies - based upon MyGPT's unique features - for configuring an effective tool with limited data. The findings suggest that, despite a few limitations, a custom GPT developed using these specific strategies can serve as a useful coding assistant by generating coding suggestions.
Study at a glance
- Question: How can researchers configure an effective, low-cost custom GPT to code classroom dialogue using limited data?
- Setting and data: Transcripts of 1,386 turns from three primary school lessons (math and English) in England, drawn from a 144-lesson dataset.
- Method: Design-based research combined with a controlled variable evaluation comparing AI coding to human annotations across different example sample sizes.
Why it matters
Analyzing classroom dialogue is labor-intensive and cognitively demanding. Training large models is often impractical with small datasets, leaving a gap between experimental AI coding and realistic implementation in everyday educational settings.
Key findings
- Targeted prompting strategies, including decision trees and iterative feedback, achieved 64.9% overall turn-level agreement with human coders, approaching the desired 70% reliability benchmark.
- A baseline model given only basic codebook definitions performed poorly, with precision ranging from 6.7% to 38.2% across most dialogue categories.
- Increasing training examples improved precision from 12 to 120 examples, but increasing further to 500 examples produced diminishing returns or declines due to token limits.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
What is the application?
Who is the user?
Who age?
Why use AI?
Study design
