Date
Publisher
arXiv
In English education tutoring, teacher feedback is essential for guiding
students. Recently, AI-based tutoring systems have emerged to assist teachers;
however, these systems require high-quality and large-scale teacher feedback
data, which is both time-consuming and costly to generate manually. In this
study, we propose FEAT, a cost-effective framework for generating teacher
feedback, and have constructed three complementary datasets: (1) DIRECT-Manual
(DM), where both humans and large language models (LLMs) collaboratively
generate high-quality teacher feedback, albeit at a higher cost; (2)
DIRECT-Generated (DG), an LLM-only generated, cost-effective dataset with lower
quality;, and (3) DIRECT-Augmented (DA), primarily based on DG with a small
portion of DM added to enhance quality while maintaining cost-efficiency.
Experimental results showed that incorporating a small portion of DM (5-10%)
into DG leads to superior performance compared to using 100% DM alone.
Study at a glance
- Question: Can AI models automatically generate and rank teacher feedback data as effectively as human annotators?
- Setting and data: Reading comprehension tasks for grades 1–4 from MCTest and tutoring dialogues from the DIRECT dataset.
- Method: Machine learning model training and evaluation using five ranking algorithms across simulated tutoring preference datasets.
Why it matters
AI-based tutoring systems need large amounts of high-quality teacher feedback to guide students, but generating and ranking feedback data manually is costly and time-consuming.
Key findings
- Models trained on AI-generated data supplemented with only 5% to 10% human-annotated data outperformed models trained entirely on human-annotated data.
- Ranking models trained solely on automated AI-generated feedback achieved competitive performance compared to models trained on human-curated datasets.
- Incorporating five educational feedback criteria rather than two essential criteria generally improved ranking model performance, with gains up to 11.41%.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
What is the application?
Who is the user?
Who age?
Why use AI?
Study design
