Large language models (LLMs) can now generate physics practice problems in real time, yet the educational value of these items hinges on rapid, reliable post-generation vetting. In this exploratory study, we investigated which automated checks are both technically feasible and pedagogically meaningful when exercises are produced on demand within a chatbot interface. A cohort of 34 introductory-physics students generated and attempted 543 practice problems during exam preparation. Each item was labeled by an expert on a wide range of quality attributes and presented to the learners in pairs to record their preference. We then (i) benchmarked three commodity LLMs as ``judges'' against the expert labels, (ii) quantified which attributes predict student choice via random-forest models, and (iii) triangulated these results with free-form exit surveys. Only a small subset of the original metric items proved necessary to reliably address student preferences either directly or by proxy. The study demonstrates that scalable formative assessment does not require exhaustive scoring: a carefully curated core of structural and learner-visible checks is sufficient to ensure both technical soundness and user appeal. The findings provide a practical blueprint for deploying real-time, AI-generated practice in physics and other quantitative disciplines.
- Question: Which automated checks can reliably evaluate the quality of AI-generated practice problems for students?
- Setting and data: 34 introductory-physics students preparing for exams, who generated and attempted 543 practice problems.
- Method: Compares AI evaluations against expert ratings, models student choices (random-forest models), and analyzes exit surveys.
Large language models can generate physics practice problems in real time, yet their educational value hinges on rapid, reliable post-generation vetting. On-demand exercises require automated checks that are both technically feasible and pedagogically meaningful.
- For 34 introductory physics students attempting 543 AI-generated exercises, only a small subset of quality measures was needed to reliably capture student preferences.
- A curated core of structural and student-visible checks is sufficient to ensure generated practice problems are technically sound and appealing to learners.
- Scalable evaluation of AI-generated practice items does not require exhaustive scoring across every potential quality attribute.
Summary drafted with AI from the paper’s abstract and may contain errors. See the full paper for details.
