Date
Publisher
arXiv
The large-scale digitization of educational assessment has made the continuous oversight of item banks both essential and complex. This paper presents Analytics for Quality Assurance for Item Pools (AQuAP), a dashboard environment for monitoring item quality and item bank health. AQuAP supports the operational implementation of the large scale item generation procedures for high-stakes tests as included in the Item Factory, a framework for automated and human-supported test development. The paper describes AQuAP in relationship with the process of item development, outlines the broader metric framework for item-pool quality assurance, and highlights the Effective Bank Size (EBS) as one central indicator of pool vitality. EBS quantifies how many independent test sessions can be constructed before content repetition occurs and, when coupled with exposure and usage metrics, provides insight into item bank security, diversity, and efficiency. We further introduce bank-health metrics, such as maximum exposure, maximum conditional exposure, adjusted effective bank size, and the rarely-administered fraction, all of which extend this picture of item utilization. AQuAP illustrates how operational analytics can translate psychometric concepts into quality assurance tools for high-volume, AI-enabled testing programs. This work is illustrated with the Duolingo English Test (DET) processes.
Study at a glance
- Question: How can operational analytics monitor item quality and item bank health in automated, adaptive testing systems?
- Setting and data: Operational data and test items from the Duolingo English Test (DET), including reading passages generated in 2023.
- Method: System development and metric framework formulation illustrated with operational data.
Why it matters
Automated item generation and adaptive delivery at scale create vulnerabilities related to item reuse, overexposure, and parameter drift, making continuous item bank oversight essential.
Key findings
- The Effective Bank Size quantifies unique test sessions possible before content repeats, establishing that values below 10 indicate constrained banks while values above 50 indicate robust security.
- Adjusting Effective Bank Size using Shannon entropy accounts for uneven item selection, revealing operational capacity loss when adaptive algorithms repeatedly administer a small subset of items.
- Complementary metrics, including maximum exposure, maximum conditional exposure, and rarely-administered fractions, identify localized overexposure within ability strata and underutilized items that aggregate measures miss.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
What is the application?
Who is the user?
Who age?
Why use AI?
Study design
