Date
Publisher
arXiv
Producing large volumes of high-quality, timely feedback poses significant challenges to instructors. To address this issue, automation technologies particularly Large Language Models (LLMs) show great potential. However, current LLM-based research still shows room for improvement in terms of feedback quality. Our study proposed a multi-agent approach performing ígeneration, evaluation, and regeneration (G-E-RG) to further enhance feedback quality. In the first-generation phase, six methods were adopted, combining three feedback theoretical frameworks and two prompt methods: zero-shot and retrieval-augmented generation with chain-of-thought (RAG_CoT). The results indicated that, compared to first-round feedback, G-E-RG significantly improved final feedback across six methods for most dimensions. Specifically:(1) Evaluation accuracy for six methods increased by 3.36% to 12.98% (p < 0.001); (2) The proportion of feedback containing four effective components rose from an average of 27.72% to an average of 98.49% among six methods, sub-dimensions of providing critiques, highlighting strengths, encouraging agency, and cultivating dialogue also showed great enhancement (p < 0.001); (3) There was a significant improvement in most of the feature values (p < 0.001), although some sub-dimensions (e.g., strengthening the teacher-student relationship) still require further enhancement; (4) The simplicity of feedback was effectively enhanced (p < 0.001) for three methods.
Study at a glance
- Question: Can an AI system that drafts, evaluates, and revises its own feedback improve overall feedback quality?
- Setting and data: Not stated in the abstract
- Method: Compares initial AI-generated feedback to feedback refined through an automated generation, evaluation, and regeneration process across six methods
Why it matters
Producing large volumes of high-quality, timely feedback poses significant challenges to instructors. While large language models show great potential to address this issue, current approaches still show room for improvement in feedback quality.
Key findings
- The proportion of AI feedback containing four effective supportive components rose from an average of 27.72% to 98.49% following automated evaluation and regeneration.
- Automated evaluation and revision increased AI feedback accuracy by 3.36% to 12.98% across all six tested generation methods.
- The multi-step AI process significantly improved feedback simplicity for three methods, though strengthening teacher-student relationships still required further improvement.
Summary drafted with AI from the paper’s abstract and may contain errors. See the full paper for details.
What is the application?
Who age?
Why use AI?
