Date
AI Overview
Why it matters
Understanding how well large language models can mark open-text responses to short answer questions can show whether they can reliably support low-stakes formative assessment in real-world K-12 education.
Study at a glance
- Question: How accurately can large language models mark student short answer questions across different subjects and grade levels?
- Setting and data: Student answers across Science and History for ages 5-16, drawn from the Carousel quizzing platform.
- Method: Experimental evaluation testing combinations of AI model versions and prompt engineering strategies against human markers.
- AI tools: GPT-4
Key findings
- GPT-4 with basic few-shot prompting scored student responses with an agreement score of 0.70 Kappa, performing very close to human-level performance at 0.75.
- The proximity to human markers held across different subject areas, specifically Science and History, and grade levels spanning ages 5 to 16.
- Results suggest that large language models could be a valuable tool for supporting low-stakes formative assessment tasks in K-12 education.
Summary drafted with AI from the paper’s abstract and may contain errors. See the full paper for details.
This paper presents reports on a series of experiments with a novel dataset evaluating how well Large Language Models (LLMs) can mark (i.e. grade) open text responses to short answer questions, Specifically, we explore how well different combinations of GPT version and prompt engineering strategies performed at marking real student answers to short answer across different domain areas (Science and History) and grade-levels (spanning ages 5-16) using a new, never-used-before dataset from Carousel, a quizzing platform. We found that GPT-4, with basic few-shot prompting performed well (Kappa, 0.70) and, importantly, very close to human-level performance (0.75). This research builds on prior findings that GPT4 could reliably score short answer reading comprehension questions at a performance-level very close to that of expert human raters. The proximity to human-level performance, across a variety of subjects and grade levels suggests that LLMs could be a valuable tool for supporting low-stakes formative assessment tasks in K-12 education and has important implications for real-world education delivery.
Research Categories
What is the application?
Who is the user?
Why use AI?
Study design
AI tools
Datasets
Publication Source
Publisher
ACM Digital Library
