Search and Filter

Feanel: A Benchmark For Fine-Grained Error Analysis In K-12 English Writing

Date

AI Overview

Why it matters

Current writing tools focus on surface corrections or coarse scores, leaving language models' ability to provide fine-grained, interpretable educational feedback for K-12 writing underexplored.

Study at a glance
  • Question: How accurately can large language models classify, rate, and explain specific writing errors in K-12 student essays?
  • Setting and data: 1,000 essays (8,676 errors) from elementary students aged 9-11 globally and secondary students aged 12-18 from the Chinese TECCL Corpus.
  • Method: Benchmark evaluation comparing multiple large language models and human teachers across zero-shot and one-shot settings.
  • AI tools: GPT-4o, o1, o3, Gemini-2.5-pro, DeepSeek-R1, Claude-3.7-Sonnet
Key findings
  • Human teachers outperformed all evaluated language models in error classification and explanation, especially under the zero-shot setting where models lacked detailed instructions.
  • Classifying errors was more difficult in elementary school essays than secondary school essays, with model accuracy typically 2 to 6 percentage points lower.
  • Providing detailed error definitions and examples substantially improved model performance across error classification, severity rating, and explanation quality.

Summary drafted with AI from the full paper and may contain errors. See the full paper for details.

Large Language Models (LLMs) have transformed artificial intelligence, offering profound opportunities for educational applications. However, their ability to provide fine-grained educational feedback for K-12 English writing remains underexplored. In this paper, we challenge the error analysis and pedagogical skills of LLMs by introducing the problem of Fine-grained Error Analysis for English Learners and present the Fine-grained Error ANalysis for English Learners (FEANEL) Benchmark. The benchmark comprises 1,000 essays written by elementary and secondary school students, and a well-developed English writing error taxonomy. Each error is annotated by language education experts and categorized by type, severity, and explanatory feedback, using a part-of-speech-based taxonomy they co-developed. We evaluate state-of-the-art LLMs on the FEANEL Benchmark to explore their error analysis and pedagogical abilities. Experimental results reveal significant gaps in current LLMs' ability to perform fine-grained error analysis, highlighting the need for advancements in particular methods for educational applications.