Search and Filter

Confusion-Aware Rubric Optimization For Llm-Based Automated Grading

Date

AI Overview

Why it matters

Scoring open-ended student work creates a heavy grading bottleneck for teachers. Current automated prompt optimization methods mix different errors together, causing conflicting instructions that weaken large language model grading precision.

Study at a glance
  • Question: Can separating specific grading error patterns improve the accuracy and efficiency of automated AI rubric optimization?
  • Setting and data: High school physics responses (252 samples), elementary science responses (199-434 samples), and teacher education mathematics assessments (224-236 samples).
  • Method: Algorithm design and empirical evaluation comparing prompt optimization frameworks (CARO, GradeOpt, and naive baseline).
  • AI tools: CARO, GradeOpt, GPT-4o-mini, SBERT
Key findings
  • CARO increased overall grading accuracy to 0.78, compared to 0.70 for GradeOpt and 0.51 for naive prompting.
  • Across all datasets, CARO achieved an overall average Cohen's kappa of 0.56, outperforming GradeOpt (0.47) and naive prompting (0.22).
  • Targeted error mode optimization reduced language model API calls by 80 percent and computational costs by 60 percent compared to GradeOpt.

Summary drafted with AI from the full paper and may contain errors. See the full paper for details.

Accurate and unambiguous guidelines are critical for large language model (LLM) based graders, yet manually crafting these prompts is often sub-optimal as LLMs can misinterpret expert guidelines or lack necessary domain specificity. Consequently, the field has moved toward automated prompt optimization to refine grading guidelines without the burden of manual trial and error. However, existing frameworks typically aggregate independent and unstructured error samples into a single update step, resulting in "rule dilution" where conflicting constraints weaken the model's grading logic. To address these limitations, we introduce Confusion-Aware Rubric Optimization (CARO), a novel framework that enhances accuracy and computational efficiency by structurally separating error signals. CARO leverages the confusion matrix to decompose monolithic error signals into distinct modes, allowing for the diagnosis and repair of specific misclassification patterns individually. By synthesizing targeted "fixing patches" for dominant error modes and employing a diversity-aware selection mechanism, the framework prevents guidance conflict and eliminates the need for resource-heavy nested refinement loops. Empirical evaluations on teacher education and STEM datasets demonstrate that CARO significantly outperforms existing SOTA methods. These results suggest that replacing mixed-error aggregation with surgical, mode-specific repair yields robust improvements in automated assessment scalability and precision.