Optimizing In-Context Demonstrations For Llm-Based Automated Grading

Yucheng Chu, Michigan State University

Hang Li, Michigan State University

Kaiqi Yang, Michigan State University

Yasemin Copur-Gencturk, University of Southern California

Kevin C. Haudek, Michigan State University

Joseph Krajcik, Michigan State University

Jiliang Tang, Michigan State University

Date

02/2026

AI Overview

Why it matters

Automated grading of open-ended student responses offers scalable feedback, but large language models struggle with consistency, sensitive exemplar selection, and subtle rubric boundaries between adjacent score levels.

Study at a glance

Question: Can optimizing contrastive few-shot examples and generated rationales improve automated grading accuracy and resolve borderline score cases?
Setting and data: High school science and mathematics teacher education datasets: 314 physical science responses, 163-184 chemistry responses, and 229-236 teacher education responses.
Method: Model evaluation optimizing exemplar selection and rationale generation using Bayesian optimization with contrastive operators.
AI tools: GPT-4o-mini, text-embedding-3-small, all-MiniLM-L6-v2, GUIDE

Key findings

GUIDE outperformed baseline grading methods across all three datasets, achieving grading accuracies of 0.92 in physics, 0.80 in chemistry, and 0.71 in teacher education.
The method substantially lowered adjacent score error rates to 0.08 in physics, 0.20 in chemistry, and 0.28 in teacher education without increasing non-adjacent errors.
Direct comparison against the BRIDGE optimization baseline showed higher accuracy and reduced boundary errors across all evaluated educational datasets.

Summary drafted with AI from the full paper and may contain errors. See the full paper for details.

Automated assessment of open-ended student responses is a critical capability for scaling personalized feedback in education. While large language models (LLMs) have shown promise in grading tasks via in-context learning (ICL), their reliability is heavily dependent on the selection of few-shot exemplars and the construction of high-quality rationales. Standard retrieval methods typically select examples based on semantic similarity, which often fails to capture subtle decision boundaries required for rubric adherence. Furthermore, manually crafting the expert rationales needed to guide these models can be a significant bottleneck. To address these limitations, we introduce GUIDE (Grading Using Iteratively Designed Exemplars), a framework that reframes exemplar selection and refinement in automated grading as a boundary-focused optimization problem. GUIDE operates on a continuous loop of selection and refinement, employing novel contrastive operators to identify "boundary pairs" that are semantically similar but possess different grades. We enhance exemplars by generating discriminative rationales that explicitly articulate why a response receives a specific score to the exclusion of adjacent grades. Extensive experiments across datasets in physics, chemistry, and pedagogical content knowledge demonstrate that GUIDE significantly outperforms standard retrieval baselines. By focusing the model's attention on the precise edges of rubric, our approach shows exceptionally robust gains on borderline cases and improved rubric adherence. GUIDE paves the way for trusted, scalable assessment systems that align closely with human pedagogical standards.