Search and Filter

Autorubric: A Unified Framework For Rubric-Based Llm Evaluation

Date

AI Overview

Why it matters

LLM judges for non-verifiable tasks remain vulnerable to biases and inconsistency. Rubrics add consequential design choices, yet existing methods remain scattered across papers and partial implementations, forcing researchers to repeatedly rebuild evaluation machinery.

Study at a glance
  • Question: Can a unified open-source framework make rubric-based LLM evaluations explicit, auditable, and effective across varied tasks?
  • Setting and data: 1,240 college chemistry answers, 65 expert-curated research questions, and 100 synthetic chatbot conversations, accessed February–March 2026.
  • Method: Benchmark evaluations, configuration sensitivity comparisons, and prompt-revision and reinforcement learning applications (Model or benchmark evaluation).
  • AI tools: Autorubric, Gemini-3-Flash, Claude Sonnet-4.5, GPT-5.4-nano, LLaMA-3.1-8B, Qwen/Qwen3-4B-Instruct
Key findings
  • On college chemistry grading, 5-shot calibration improved Gemini-3-Flash rubric entailment accuracy from 78.0% to 80.7% on held-out student responses.
  • Evaluating deep research systems revealed that Gemini-3-Flash was 5 to 6 times cheaper than Claude Sonnet-4.5 while maintaining the same overall system ranking.
  • Model family was the largest source of evaluation variation on CHARM-100, where default exact accuracy ranged from 60.4% down to 19.4%.

Summary drafted with AI from the full paper and may contain errors. See the full paper for details.

Rubric-based evaluation with large language models (LLMs) has become standard practice for assessing text generation at scale, yet the underlying techniques are scattered across papers with inconsistent terminology and partial solutions. We present a unified framework: each identified technique is paired with its realization in Autorubric, an open-source Python framework proposed in this paper. Autorubric supports binary, ordinal, and nominal criteria with configurable weights; single-judge and multi-judge ensemble evaluation with majority, weighted, unanimous, and any-vote aggregation; few-shot calibration with verdict-balanced sampling; and mitigations for position bias (option shuffling), verbosity bias (length penalties), and criterion conflation (per-criterion atomic evaluation with natural language explanations). The framework provides reliability metrics drawn from psychometrics (Cohen's $κ$, weighted $κ$, correlation coefficients, and distribution-level tests) alongside production infrastructure including response caching, checkpointing with resumable runs, multi-provider rate limiting, and cost tracking. We evaluate Autorubric on three benchmarks spanning educational assessment, deep research evaluation, and chatbot quality assessment, demonstrating that it produces results consistent with published benchmarks while exercising the framework's key capabilities: per-criterion binary evaluation with few-shot calibration (RiceChem), multi-judge ensemble evaluation across judge models (ResearcherBench), and mixed criterion types combining binary, ordinal, and nominal scales (CHARM-100). We also contribute CHARM-100, a 100-sample chatbot evaluation dataset with per-sample ground truth labels across all three criterion types, designed to stress-test rubric evaluation frameworks on heterogeneous criteria.