Despite growing interest in using LLMs to generate feedback on students' writing, little is known about how students respond to AI-mediated versus human-provided feedback. We address this gap through a randomized controlled trial in a large introductory economics course (N=354), where we introduce and deploy FeedbackWriter - a system that generates AI suggestions to teaching assistants (TAs) while they provide feedback on students' knowledge-intensive essays. TAs have the full capacity to adopt, edit, or dismiss the suggestions. Students were randomly assigned to receive either handwritten feedback from TAs (baseline) or AI-mediated feedback where TAs received suggestions from FeedbackWriter. Students revise their drafts based on the feedback, which is further graded. In total, 1,366 essays were graded using the system. We found that students receiving AI-mediated feedback produced significantly higher-quality revisions, with gains increasing as TAs adopted more AI suggestions. TAs found the AI suggestions useful for spotting gaps and clarifying rubrics.
- Question: Does AI-assisted human feedback improve college students' essay revisions and learning compared to human-only feedback?
- Setting and data: Introductory economics course at a US public university, January–April 2025, 354 students, 11 teaching assistants, and 1,366 essays.
- Method: Randomized controlled trial with counterbalanced assignments, mixed-effects linear regression, and semi-structured interviews (Mixed methods).
Providing targeted feedback on knowledge-intensive student writing requires substantial effort. While AI feedback can help, unvetted models make errors, and little is known about how students respond to AI-assisted human feedback compared to human-only feedback.
- Students receiving AI-mediated feedback produced significantly higher-quality revisions than those receiving human-only feedback (effect size d = 0.50), equivalent to moving from the 50th to 70th percentile.
- Revision gains were larger when teaching assistants adopted more AI-suggested constructive feedback, but student scores on a post-test did not differ significantly between conditions.
- AI-assisted feedback was rated significantly higher in actionability, tone and supportiveness, and promotion of independent learning compared to human-only feedback.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
