Date
Publisher
arXiv
This study examines feedback in English as a Foreign Language (EFL) writing contexts, focusing on written corrective feedback (WCF). Large language models (LLMs) can provide WCF at scale, but aligning them with pedagogical best practices remains an ongoing challenge. WCF meeting criteria like factuality or relevance may still be unsuitable for learning contexts, highlighting the need for extrinsic evaluation based on the learner's perspective. We deployed WCF systems in a university-level EFL class with nearly 2,000 students, collecting over 20,000 drafts. We evaluated the generated WCF from two perspectives: intrinsic evaluation by experienced English teachers using a rubric, and extrinsic evaluation via student feedback and engagement metrics. Results revealed low alignment between teacher expert ratings and student feedback. These findings suggest that traditional expert evaluation alone may not fully capture WCF's usability or helpfulness from the learner's perspective, highlighting the importance of learner-centered evaluation frameworks for AI-based applications in language education.
Study at a glance
- Question: How do teacher evaluations of AI-generated writing feedback compare with student engagement and ratings?
- Setting and data: Online college English course, 1,899 second-year undergraduates, 20,255 essay drafts, and 581 sampled feedback comments evaluated by four teachers.
- Method: Counterbalanced repeated measures design comparing three feedback systems, combined with expert rubric ratings and student engagement logs.
Why it matters
Large language models can generate writing feedback at scale, but expert evaluations may fail to capture whether learners find it helpful or usable.
Key findings
- Expert teacher ratings of feedback quality did not significantly align with students' thumbs-up or thumbs-down ratings (p=0.0625).
- Teachers rated both large language model systems higher than a template baseline (mean overall quality scores 4.51 and 4.36 versus 3.14 out of 5).
- Nearly half of students (49.76%) never viewed feedback, but those who engaged submitted mostly positive ratings, with 92.35% being likes.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
What is the application?
Who is the user?
Who age?
Why use AI?
Study design
