Date
Publisher
arXiv
We present a method for generating large numbers of isomorphic physics
problems using generative AI services such as ChatGPT, through prompt chaining
and tool use. This approach enables precise control over structural
variations-such as numeric values and spatial relations-while supporting
diverse contextual variations in the problem body. By utilizing the Python code
interpreter, the method supports automatic solution validation and simple
diagram generation, addressing key limitations in existing LLM-based methods.
We generated two example isomorphic problem banks and compared the outcome
against two simpler prompt-based approaches. Results show that prompt-chaining
produces significantly higher quality and more consistent outputs than simpler,
non-chaining prompts. We also show that GenAI services can be used to validate
the quality of the generated isomorphic problems. This work demonstrates a
promising method for efficient and scalable problem creation accessible to the
average instructor, which opens new possibilities for personalized adaptive
testing and automated content development.
Study at a glance
- Question: Can prompt-chaining and tool use reliably generate high-quality isomorphic physics problems and diagrams?
- Setting and data: University physics course context, 20 numerical problems and 26 conceptual problems generated across two problem banks.
- Method: Comparative evaluation of prompt chaining with code execution versus single-prompt generation.
Why it matters
Automated question generation with large language models often struggles to control difficulty, cognitive complexity, and answer accuracy, while generating diagrams remains a persistent obstacle.
Key findings
- Prompt chaining produced significantly higher-quality and more consistent physics problems than simpler, single-prompt approaches that missed numerical values or generated incorrect calculations.
- Using Python code execution within the prompt chain enabled precise calculation checks and successfully generated downloadable trajectory diagrams that single prompts failed to produce correctly.
- Generative AI verification detected errors, including six derivation errors in problem solutions and three equivalent distractors, effectively catching model hallucinations.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
What is the application?
Who is the user?
Who age?
Why use AI?
Study design
