Search and Filter

Assessing the Quality of AI-Generated Exams: A Large-Scale Field Study

Date

AI Overview

Why it matters

Writing customized exam questions takes instructors hours or days, but there is little evidence evaluating the psychometric quality of generative AI-created questions in real-world educational settings.

Study at a glance
  • Question: How do AI-generated exam questions compare psychometrically to human expert-designed exam questions in real college courses?
  • Setting and data: 91 US college classes (mainly STEM), Spring 2025, 1,686 students completing pre-tests and course-specific final exams.
  • Method: Field study using item response theory (two-parameter logistic hierarchical Bayesian model).
  • AI tools: o3-mini
Key findings
  • AI-generated exam questions performed comparably to expert-created standardized test questions, showing slightly higher discrimination but somewhat easier difficulty.
  • Students answered about 60% of AI-generated questions correctly, compared to 39% of standardized test questions.
  • Exams with AI-generated questions delivered higher overall test information and reliability for students of average and below-average ability compared to standardized test benchmark exams.

Summary drafted with AI from the full paper and may contain errors. See the full paper for details.

While large language models (LLMs) challenge conventional methods of teaching and learning, they present an exciting opportunity to improve efficiency and scale high-quality instruction. One promising application is the generation of customized exams, tailored to specific course content. There has been significant recent excitement on automatically generating questions using artificial intelligence, but also comparatively little work evaluating the psychometric quality of these items in real-world educational settings. Filling this gap is an important step toward understanding generative AI's role in effective test design. In this study, we introduce and evaluate an iterative refinement strategy for question generation, repeatedly producing, assessing, and improving questions through cycles of LLM-generated critique and revision. We evaluate the quality of these AI-generated questions in a large-scale field study involving 91 classes -- covering computer science, mathematics, chemistry, and more -- in dozens of colleges across the United States, comprising nearly 1700 students. Our analysis, based on item response theory (IRT), suggests that for students in our sample the AI-generated questions performed comparably to expert-created questions designed for standardized exams. Our results illustrate the power of AI to make high-quality assessments more readily available, benefiting both teachers and students.