Date
Publisher
arXiv
It is astonishing how rapidly general-purpose AI has crossed familiar
thresholds in introductory physics. Comparing outputs from successive models,
GPT-5 Thinking moves far beyond the plug-and-chug tendencies seen earlier: on a
classic elevator problem it works symbolically, notes when variables cancel,
and verifies results; attempts to prompt novice-like behavior mainly affect
tone, not method. On representation translation, the model scores 24/26 (92.3%)
on TUG-Kv4.0. In a card-sorting proxy using two of my comprehensive finals (60
items), its categories reflect solution method rather than surface features.
Solving those same exams, it attains 27/30 and 25/30, with most misses in
ruler-based ray tracing and circuit interpretation. On epistemology, five
independent CLASS runs yield 100\% favorable, indicating a simulated
expert-like stance. Framed as a "boiling frog" problem, the paper argues for a
decisive jump: retire credit-bearing unsupervised closed-response online
assessments; grade process evidence; use paper, whiteboarding; shift weight to
modeling, data, and authentic labs; require transparent, citable AI use;
rebuild problem types; and lean on research-based instruction and peer
learning. The opportunity is to foreground what AI cannot substitute for:
modeling the world, arguing from evidence, and making principled
approximations.
Study at a glance
- Question: How well can advanced artificial intelligence solve introductory physics problems and demonstrate expert-like reasoning?
- Setting and data: Introductory physics problems, two 30-item comprehensive final exams, and standard physics assessments evaluated by AI models.
- Method: Tests AI model performance across physics problem-solving tasks, final exams, card-sorting exercises, and a physics survey.
Why it matters
General-purpose AI is rapidly crossing familiar thresholds in introductory physics, creating a boiling-frog problem for physics education.
Key findings
- In introductory physics, advanced AI solved 27 of 30 and 25 of 30 exam questions, struggling mainly with ray tracing and circuits.
- The AI worked through problems symbolically and verified results, maintaining expert-level methods even when prompted to act like a novice learner.
- The AI grouped exam problems by solution methods rather than surface features and consistently simulated an expert-like perspective on learning physics.
Summary drafted with AI from the paper’s abstract and may contain errors. See the full paper for details.
What is the application?
Who is the user?
Who age?
Why use AI?
Study design
