Date
Publisher
arXiv
The discipline of physics stands as a cornerstone of human intellect, driving the evolution of technology and deepening our understanding of the fundamental principles of the cosmos. Contemporary literature includes some works centered on the task of solving physics problems - a crucial domain of natural language reasoning. In this paper, we evaluate the performance of frontier LLMs in solving physics problems, both mathematical and descriptive. We also employ a plethora of inference-time techniques and agentic frameworks to improve the performance of the models. This includes the verification of proposed solutions in a cumulative fashion by other, smaller LLM agents, and we perform a comparative analysis of the performance that the techniques entail. There are significant improvements when the multi-agent framework is applied to problems that the models initially perform poorly on. Furthermore, we introduce a new evaluation benchmark for physics problems, ${\rm P{\small HYSICS}E{\small VAL}}$, consisting of 19,609 problems sourced from various physics textbooks and their corresponding correct solutions scraped from physics forums and educational websites. Our code and data are publicly available at https://github.com/areebuzair/PhysicsEval.
Study at a glance
- Question: Can multi-agent verification and inference-time techniques improve large language model reasoning on physics problems?
- Setting and data: 1,962 physics problems sampled from the 19,609-problem PHYSICSEVAL benchmark, covering high school to university textbooks and online educational forums.
- Method: Benchmarking evaluation comparing baseline, self-refinement, single-agent review, and multi-agent verification using a rubric-based automated evaluator.
Why it matters
Large language models often struggle to reason accurately through physics problems. Current mathematical benchmarks are proving insufficient to distinguish between pattern matching and genuine scientific reasoning grounded in physical principles.
Key findings
- A multi-agent review framework achieved the highest average proficiency scores across easy (91.78), medium (81.48), and hard (68.23) physics problems compared to baseline performance.
- Gains from multi-agent verification amplified on hard problems, improving Phi-4-reasoning-plus by 7.5 points and QwQ-32B by 7.1 points over their baselines.
- Self-refinement without external guidance harmed performance for models like Gemma 3 27B and Llama 4 Maverick, leading to lower scores across all difficulty levels.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
What is the application?
Who age?
Why use AI?
Study design
