Date
Publisher
arXiv
LLMs now help students plan, draft, and revise essays. Educational assessment therefore faces a basic question: how should student and LLM writing be compared? Rubrics assign points to content, organization, and expression. Their total can still hide which criteria drive the comparison, where ratings approach the maximum, and where readers disagree. We therefore conducted a secondary, post hoc audit of a Korean writing study with source-informed scoring. Three Korean language and literature majors used a 16-criterion, 100-point rubric to score six student essays and 24 essays from four LLMs prompted for three school levels. After source disclosure, they helped develop the rubric and, according to the protocol, scored the Phase 2 essays without gold labels while recording source judgments. The pooled LLM mean exceeds the student mean by 18.35 points. Orthographic norms and genre-appropriate register contribute 8.19 points, or 44.7% of the gap. All 72 LLM ratings reach the orthographic maximum, and 70 reach the register maximum. The student source mean is highest or tied highest on both creativity criteria and natural Korean phrasing, where reader agreement is weak. Criterion-level auditing therefore offers a deeper account of human and LLM writing than the total alone.
Study at a glance
- Question: Which specific rubric criteria drive the scoring gap between student and large language model essays?
- Setting and data: South Korea; 30 Korean argumentative essays (6 student essays and 24 LLM essays across elementary, middle, and high school levels) scored by 3 undergraduate readers.
- Method: Descriptive analysis and score decomposition of a secondary, post hoc reader evaluation audit.
Why it matters
Overall writing rubric scores can hide which specific criteria drive comparisons between student and AI essays, where ratings reach ceilings, and where human raters disagree.
Key findings
- Large language model essays outscored student essays by an average of 18.35 points across a 100-point rubric, exceeding student averages across all four evaluated models.
- Nearly half of the score gap (44.7%) came from just two formal criteria—orthographic norms and genre-appropriate register—where AI ratings clustered near the maximum.
- Student essays had the highest or tied-highest scores on content creativity, expression creativity, and natural Korean phrasing, where agreement among the human readers was weak.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
What is the application?
Who is the user?
Why use AI?
Study design
