Search and Filter

Llm-Based Automated Grading With Human-In-The-Loop

Date

AI Overview

Why it matters

Automated grading methods struggle with jargon in rubrics and complex language when fully automated. Incorporating human feedback can help refine rubrics and ensure controllable grading standards in education.

Study at a glance
  • Question: Can incorporating human-in-the-loop feedback improve automated rubric-based grading accuracy in large language models?
  • Setting and data: 1,376 open-ended responses from teachers across six mathematics teaching knowledge assessment questions collected through a national research study.
  • Method: Model evaluation comparing a human-in-the-loop rubric optimization framework against baseline grading methods.
  • AI tools: GradeHITL, GradeOpt, GPT-4o, RoBERTa, SBERT
Key findings
  • GradeHITL consistently outperformed fully automated rubric optimization and baseline grading models across all six tested questions.
  • A reinforcement learning retriever for human questions and answers outperformed random and semantic similarity retrieval methods across all test questions.
  • All four tested large language model grading approaches consistently surpassed non-language model baselines like RoBERTa and SBERT on the grading tasks.

Summary drafted with AI from the full paper and may contain errors. See the full paper for details.

The rise of artificial intelligence (AI) technologies, particularly large language models (LLMs), has brought significant advancements to the field of education. Among various applications, automatic short answer grading (ASAG), which focuses on evaluating open-ended textual responses, has seen remarkable progress with the introduction of LLMs. These models not only enhance grading performance compared to traditional ASAG approaches but also move beyond simple comparisons with predefined "golden" answers, enabling more sophisticated grading scenarios, such as rubric-based evaluation. However, existing LLM-powered methods still face challenges in achieving human-level grading performance in rubric-based assessments due to their reliance on fully automated approaches. In this work, we explore the potential of LLMs in ASAG tasks by leveraging their interactive capabilities through a human-in-the-loop (HITL) approach. Our proposed framework, GradeHITL, utilizes the generative properties of LLMs to pose questions to human experts, incorporating their insights to refine grading rubrics dynamically. This adaptive process significantly improves grading accuracy, outperforming existing methods and bringing ASAG closer to human-level evaluation.