Date
Publisher
arXiv
We present a method for generating large numbers of isomorphic physics
problems using ChatGPT through prompt chaining and tool use. This approach
enables precise control over structural variations-such as numeric values and
spatial relations-while supporting diverse contextual variations in the problem
body. By utilizing the Python code interpreter, the method supports automatic
solution validation and simple diagram generation, addressing key limitations
in existing LLM-based methods. We generated two example isomorphic problem
banks and compared the outcome against simpler prompt-based approaches. Results
show that prompt-chaining produces significantly higher quality and more
consistent outputs than simpler, non-chaining prompts. This work demonstrates a
promising method for efficient problem creation accessible to the average
instructor, which opens new possibilities for personalized adaptive testing and
automated content development.
Study at a glance
- Question: Can prompt chaining and code execution tools reliably generate high-quality isomorphic physics problems and diagrams?
- Setting and data: University of Central Florida; two generated physics problem banks containing 46 total isomorphic questions administered on a midterm exam.
- Method: Design and evaluation of a prompt-chaining and code execution framework compared to single-prompt and simple-prompt generation.
Why it matters
Automated question generation reduces the time and cost of item creation. However, language models often struggle to control problem difficulty, hallucinate calculation answers, and fail to generate diagrams.
Key findings
- Prompt chaining combined with the Python code interpreter produced consistent physics problems with verified solutions and usable diagrams, whereas simpler prompts failed or produced calculation hallucinations.
- Using a single prompt neglected instructions to generate numerical values entirely, and simple prompts frequently hallucinated incorrect numerical answers without executing code.
- Generative AI successfully acted as a quality validator, identifying and correcting formula derivation errors in 6 of 20 computational problems and detecting equivalent distractors in 3 conceptual problems.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
What is the application?
Who is the user?
Who age?
Why use AI?
Study design
