Search and Filter

Repository Feedback

Your feedback helps us improve the repository's content relevance and usability. Please share your thoughts to help us better serve researchers and practitioners.

Submit feedback

Submit a research study

Contribute to the repository:

Add a paper

ChatGPT-4 with Code Interpreter can be used to solve introductory college-level vector calculus and electromagnetism problems

Tanuj Kumar, University of Wisconsin–Madison
Mikhail A. Kats, University of Wisconsin–Madison
Date
Publisher
arXiv
Link
http://arxiv.org/pdf/2309.08881v1
We evaluated ChatGPT 3.5, 4, and 4 with Code Interpreter on a set of college-level engineering-math and electromagnetism problems, such as those often given to sophomore electrical engineering majors. We selected a set of 13 problems, and had ChatGPT solve them multiple times, using a fresh instance (chat) each time. We found that ChatGPT-4 with Code Interpreter was able to satisfactorily solve most problems we tested most of the time -- a major improvement over the performance of ChatGPT-4 (or 3.5) without Code Interpreter. The performance of ChatGPT was observed to be somewhat stochastic, and we found that solving the same problem N times in new ChatGPT instances and taking the most-common answer was an effective strategy. Based on our findings and observations, we provide some recommendations for instructors and students of classes at this level.
Study at a glance
  • Question: How accurately can different versions of ChatGPT solve introductory college-level engineering mathematics and electromagnetism problems?
  • Setting and data: 13 undergraduate-level vector calculus and electromagnetism problems typical for sophomore electrical engineering majors; evaluated August 2023.
  • Method: Model evaluation across repeated independent trials (typically 10 trials per problem) using strict manual grading.
Why it matters

Generative AI capabilities are rapidly advancing, raising questions about whether tools with code execution can accurately solve college-level engineering-math and electromagnetism problems commonly assigned to undergraduate students.

Key findings
  • ChatGPT-4 with Code Interpreter achieved the correct consensus answer across multiple runs on all 13 tested problems, substantially outperforming earlier versions.
  • Performance improved notably from ChatGPT-3.5 to ChatGPT-4, and jumped again with Code Interpreter, which enabled accurate symbolic manipulations and numerical calculations.
  • Tested models exhibited stochastic performance, struggled to self-correct without explicit user hints, and tested custom prompts showed no measurable performance improvements.

Summary drafted with AI from the full paper and may contain errors. See the full paper for details.

What is the application?
Teaching – Instructional Materials,
Learning – Student Support
Who is the user?
Student,
Educator
Who age?
Post-Secondary
Why use AI?
Outcomes – Other Academic,
Outcomes – Differentiation
Study design
Technical – Computational
Download PDF