Date
Publisher
arXiv
This study examines Artificial Intelligence (AI)-generated physics solutions from two connected perspectives: how prompt design shapes these solutions and how students can be prepared to critique them. Using a rotational-mechanics problem, we adapted a problem-classification framework to examine prompt variations, evaluating OpenAI's o4-mini responses with the Minnesota Assessment of Problem Solving (MAPS) rubric. Well-specified prompts improved solution completeness; underspecified and multimodal prompts exposed weaknesses in physics reasoning and correctness. In the student-evaluation phase, 24 introductory physics lab groups evaluated an o4-mini solution to this problem after either independently solving a related problem or critiquing its AI-generated solution with MAPS-based reflection questions. Problem-solving-only groups exhibited uncritical or misconception-based critiques; MAPS-guided groups identified more expert-aligned issues, including skipped numerical procedures and undefined notation. Together, our findings contribute to physics education research by showing how AI-generated solutions can ground both model-reasoning benchmarks and improved student critique of that reasoning through MAPS-based reflection.
Why it matters
Understanding AI-generated physics solutions requires examining two connected areas: how prompt design shapes these solutions and how students can be prepared to critique them.
Key findings
- Across 24 introductory physics lab groups, students using guided reflection questions identified more expert-aligned errors in AI solutions, including skipped steps and undefined notation.
- Students who evaluated AI solutions after only solving problems independently produced uncritical critiques or based their feedback on physics misconceptions.
- Well-specified prompts improved AI physics solution completeness, but underspecified and multimodal prompts exposed weaknesses in the tool's reasoning and correctness.
Summary drafted with AI from the paper’s abstract and may contain errors. See the full paper for details.
What is the application?
Who is the user?
Who age?
Why use AI?
Study design
