Search and Filter

Ai-Enabled Grading With Near-Domain Data For Scaling Feedback With Human-Level Accuracy

Date

AI Overview

Why it matters

Manual grading of constructed-response questions is labor-intensive, creating resource constraints for timely feedback in large classes. Leveraging data from related questions could enable automated grading with minimal new labeled data.

Study at a glance
  • Question: Can AI models accurately grade student short answers using training data from related questions?
  • Setting and data: Introductory undergraduate biology courses across multiple large public research universities in the US; 2,861 students answering three constructed-response questions.
  • Method: Model performance evaluation comparing sequential fine-tuning of BERT on related questions against zero-shot and fine-tuned GPT models.
  • AI tools: BERT, GPT 3.5 Turbo, GPT 4, GPT 4 Turbo, GPT 4o
Key findings
  • Sequential fine-tuning of BERT on related questions matched baseline performance using as little as 2.5% to 7.5% of the target question's training data.
  • Fine-tuned BERT achieved up to 91.3% accuracy on transcription and 86.9% on translation, outperforming non-fine-tuned GPT models by over 10% to 20%.
  • Fine-tuned GPT-3.5 Turbo achieved the highest overall accuracy, between 87.6% and 93.9%, but fine-tuned BERT achieved comparable accuracy using significantly fewer computational parameters.

Summary drafted with AI from the full paper and may contain errors. See the full paper for details.

Constructed-response questions are crucial to encourage generative processing and test a learner's understanding of core concepts. However, the limited availability of instructor time, large class sizes, and other resource constraints pose significant challenges in providing timely and detailed evaluation, which is crucial for a holistic educational experience. In addition, providing timely and frequent assessments is challenging since manual grading is labor intensive, and automated grading is complex to generalize to every possible response scenario. This paper proposes a novel and practical approach to grade short-answer constructed-response questions. We discuss why this problem is challenging, define the nature of questions on which our method works, and finally propose a framework that instructors can use to evaluate their students' open-responses, utilizing near-domain data like data from similar questions administered in previous years. The proposed method outperforms the state of the art machine learning models as well as non-fine-tuned large language models like GPT 3.5, GPT 4, and GPT 4o by a considerable margin of over 10-20% in some cases, even after providing the LLMs with reference/model answers. Our framework does not require pre-written grading rubrics and is designed explicitly with practical classroom settings in mind. Our results also reveal exciting insights about learning from near-domain data, including what we term as accuracy and data advantages using human-labeled data, and we believe this is the first work to formalize the problem of automated short answer grading based on the near-domain data.