Large language models (LLMs) can now generate physics practice problems in real time, yet the educational value of these items hinges on rapid, reliable post-generation vetting. In this exploratory study, we investigated which automated checks are both technically feasible and pedagogically meaningful when exercises are produced on demand within a chatbot interface. A cohort of 34 introductory-physics students generated and attempted 543 practice problems during exam preparation. Each item was labeled by an expert on a wide range of quality attributes and presented to the learners in pairs to record their preference. We then (i) benchmarked three commodity LLMs as ``judges'' against the expert labels, (ii) quantified which attributes predict student choice via random-forest models, and (iii) triangulated these results with free-form exit surveys. Only a small subset of the original metric items proved necessary to reliably address student preferences either directly or by proxy. The study demonstrates that scalable formative assessment does not require exhaustive scoring: a carefully curated core of structural and learner-visible checks is sufficient to ensure both technical soundness and user appeal. The findings provide a practical blueprint for deploying real-time, AI-generated practice in physics and other quantitative disciplines.
- Question: How does generative AI impact higher computing education ethically, societally, and across institutional policies?
- Setting and data: Higher education institutions globally; 293 reviewed research articles (2020–2025) and 21 university policy documents from multiple world regions.
- Method: Systematic literature review, documentary and criterion-based policy analysis, and conceptual framework development with dilemma analysis.
The need to incorporate ethical and societal perspectives into higher computing education's adoption of generative AI is pressing, as these technologies pose challenges regarding academic integrity, equity, bias, and data provenance.
- Most evaluated computing education research on ethics concentrated heavily on honesty, deception, and academic integrity, leaving topics like equity, power, and environmental sustainability comparatively under-researched.
- Institutional policies exist along a continuum from compliance-focused approaches emphasizing academic integrity to virtues-focused approaches promoting principled adoption, with many institutions leaving operational decisions to instructors.
- The authors developed the Ethical and Societal Impacts Framework to help educational stakeholders resolve practical dilemmas around cognitive offloading, automation versus augmentation, and disclosure of tool use.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
