Date
AI Overview
Why it matters
Writing customized exam questions takes instructors hours or days, but there is little evidence evaluating the psychometric quality of generative AI-created questions in real-world educational settings.
Study at a glance
- Question: How do AI-generated exam questions compare psychometrically to human expert-designed exam questions in real college courses?
- Setting and data: 91 US college classes (mainly STEM), Spring 2025, 1,686 students completing pre-tests and course-specific final exams.
- Method: Field study using item response theory (two-parameter logistic hierarchical Bayesian model).
- AI tools: o3-mini
Key findings
- AI-generated exam questions performed comparably to expert-created standardized test questions, showing slightly higher discrimination but somewhat easier difficulty.
- Students answered about 60% of AI-generated questions correctly, compared to 39% of standardized test questions.
- Exams with AI-generated questions delivered higher overall test information and reliability for students of average and below-average ability compared to standardized test benchmark exams.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
While large language models (LLMs) challenge conventional methods of teaching
and learning, they present an exciting opportunity to improve efficiency and
scale high-quality instruction. One promising application is the generation of
customized exams, tailored to specific course content. There has been
significant recent excitement on automatically generating questions using
artificial intelligence, but also comparatively little work evaluating the
psychometric quality of these items in real-world educational settings. Filling
this gap is an important step toward understanding generative AI's role in
effective test design. In this study, we introduce and evaluate an iterative
refinement strategy for question generation, repeatedly producing, assessing,
and improving questions through cycles of LLM-generated critique and revision.
We evaluate the quality of these AI-generated questions in a large-scale field
study involving 91 classes -- covering computer science, mathematics,
chemistry, and more -- in dozens of colleges across the United States,
comprising nearly 1700 students. Our analysis, based on item response theory
(IRT), suggests that for students in our sample the AI-generated questions
performed comparably to expert-created questions designed for standardized
exams. Our results illustrate the power of AI to make high-quality assessments
more readily available, benefiting both teachers and students.
Research Categories
What is the application?
Who is the user?
Who age?
Why use AI?
Study design
AI tools
Datasets
Publication Source
Publisher
arXiv
