Date
AI Overview
Why it matters
AI-generated writing feedback is often generic, opaque, and misaligned with user criteria, making it hard to trust or act upon. Structured rubrics can scaffold evaluation, but creating high-quality rubrics and making AI reasoning intelligible remain difficult.
Study at a glance
- Question: Can an interactive rubric-based AI system improve writing revision and user-designed rubrics compared to standard AI feedback?
- Setting and data: University students and tutors recruited online, participating in remote lab studies and writing evaluations using learner essays.
- Method: Two randomized between-subjects lab experiments, an end-to-end user study, and expert validation (Mixed methods).
- AI tools: iRULER, GPT-4.1
Key findings
- Writers using iRULER achieved higher quality score improvements in their revised essays than those receiving text-based or read-only rubric AI feedback.
- Participants using iRULER created higher-quality rubrics, reporting higher perceived helpfulness and correctness than those using baseline systems.
- Using iRULER led to larger gains in unaided writing revision skills on post-tasks than text-based LLM feedback, suggesting transferable skill learning.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
Large Language Models (LLMs) have become indispensable for evaluating writing. However, text feedback they provide is often unintelligible, generic, and not specific to user criteria. Inspired by structured rubrics in education and intelligible AI explanations, we propose iRULER following identified design guidelines to \textit{scaffold} the review process by \textit{specific} criteria, providing \textit{justification} for score selection, and offering \textit{actionable} revisions to target different quality levels. To \textit{qualify} user-defined criteria, we recursively used iRULER with a rubric-of-rubrics to iteratively \textit{refine} rubrics. In controlled experiments on writing revision and rubric creation, iRULER most improved validated LLM-judged review scores and was perceived as most helpful and aligned compared to read-only rubric and text-based LLM feedback. Qualitative findings further support how iRULER satisfies the design guidelines for user-defined feedback. This work contributes interactive rubric tools for intelligible LLM-based review and revision of writing, and user-defined rubric creation.
Research Categories
What is the application?
Who age?
Why use AI?
Datasets
Publication Source
Publisher
arXiv
