Date
Publisher
arXiv
This project examines the prospect of using AI-generated feedback as suggestions to expedite and enhance human instructors ' feedback provision. In particular, we focus on understanding the teaching assistants ' perspectives on the quality of AI-generated feedback and how they may or may not utilize AI feedback in their own workflows. We situate our work in a foundational college Economics class, which has frequent short essay assignments. We developed an LLM-powered feedback engine that generates feedback on students ' essays based on grading rubrics used by the teaching assistants (TAs). To ensure that TAs can meaningfully critique and engage with the AI feedback, we had them complete their regular grading jobs. For a randomly selected set of essays that they had graded, we used our feedback engine to generate feedback and displayed the feedback as in-text comments in a Word document. We then performed think-aloud studies with 5 TAs over 20 1-hour sessions to have them evaluate the AI feedback, contrast the AI feedback with their handwritten feedback, and share how they envision using the AI feedback if they were offered as suggestions. The study highlights the importance of providing detailed rubrics for AI to generate high-quality feedback for knowledge-intensive essays. TAs considered that using AI feedback as suggestions during their grading could expedite grading, enhance consistency, and improve overall feedback quality. We discuss the importance of decomposing the feedback generation task into steps and presenting intermediate results, in order for TAs to use the AI feedback.
Study at a glance
- Question: How do teaching assistants evaluate AI-generated essay feedback and envision using it in their grading workflows?
- Setting and data: Foundational college economics class, short student essays, and 5 teaching assistants participating in 20 one-hour sessions.
- Method: Think-aloud study evaluating AI-generated essay feedback and comparing it with handwritten feedback.
Why it matters
Using AI-generated feedback as suggestions could expedite and enhance human instructors' feedback provision. Examining this requires understanding teaching assistants' perspectives on the quality of AI feedback and how they may or may not utilize it in their workflows.
Key findings
- In a college economics course, five teaching assistants reported that AI-generated suggestions could speed up grading, increase consistency, and enhance overall feedback quality.
- Detailed grading rubrics were essential for the language model to generate high-quality feedback on knowledge-intensive essays.
- Teaching assistants needed the feedback generation process broken down into steps showing intermediate results to effectively use the AI suggestions.
Summary drafted with AI from the paper’s abstract and may contain errors. See the full paper for details.
What is the application?
Who is the user?
Who age?
Why use AI?
