Date
AI Overview
Why it matters
Ensuring assessment items align with content standards is essential for fairness and instructional relevance, but manual reviews are costly and slow. Scalable automated tools are urgently needed to audit large item banks and support adaptive testing.
Study at a glance
- Question: Can large language models accurately check and match test items to educational content standards?
- Setting and data: Kindergarten through Grade 5 math and reading assessment items, comprising 3,011 validated items and 12,044 item–skill pairs.
- Method: Model evaluation across binary classification, open-set matching, and retrieval-augmented classification tasks.
- AI tools: GPT-3.5 Turbo, GPT-4o-mini, GPT-4o, all-MiniLM-L6-v2
Key findings
- GPT-4o-mini correctly identified whether items were aligned in 83% to 94% of cases, outperforming GPT-3.5 Turbo across all grades and subjects.
- Model accuracy dropped markedly when items were only slightly misaligned within the same domain, falling as low as 45% in math and 49% in reading.
- Pre-filtering candidate skills using sentence embeddings improved open-set classification, placing the correct skill in the top five suggestions over 95% of the time.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
As educational systems evolve, ensuring that assessment items remain aligned with content standards is essential for maintaining fairness and instructional relevance. Traditional human alignment reviews are accurate but slow and labor-intensive, especially across large item banks. This study examines whether Large Language Models (LLMs) can accelerate this process without sacrificing accuracy. Using over 12,000 item-skill pairs in grades K-5, we tested three LLMs (GPT-3.5 Turbo, GPT-4o-mini, and GPT-4o) across three tasks that mirror real-world challenges: identifying misaligned items, selecting the correct skill from the full set of standards, and narrowing candidate lists prior to classification. In Study 1, GPT-4o-mini correctly identified alignment status in approximately 83-94% of cases, including subtle misalignments. In Study 2, performance remained strong in mathematics but was lower for reading, where standards are more semantically overlapping. Study 3 demonstrated that pre-filtering candidate skills substantially improved results, with the correct skill appearing among the top five suggestions more than 95% of the time. These findings suggest that LLMs, particularly when paired with candidate filtering strategies, can significantly reduce the manual burden of item review while preserving alignment accuracy. We recommend the development of hybrid pipelines that combine LLM-based screening with human review in ambiguous cases, offering a scalable solution for ongoing item validation and instructional alignment.
Research Categories
What is the application?
Who is the user?
Who age?
Why use AI?
Study design
AI tools
Publication Source
Publisher
arXiv
