Date
Publisher
arXiv
We evaluated ChatGPT 3.5, 4, and 4 with Code Interpreter on a set of
college-level engineering-math and electromagnetism problems, such as those
often given to sophomore electrical engineering majors. We selected a set of 13
problems, and had ChatGPT solve them multiple times, using a fresh instance
(chat) each time. We found that ChatGPT-4 with Code Interpreter was able to
satisfactorily solve most problems we tested most of the time -- a major
improvement over the performance of ChatGPT-4 (or 3.5) without Code
Interpreter. The performance of ChatGPT was observed to be somewhat stochastic,
and we found that solving the same problem N times in new ChatGPT instances and
taking the most-common answer was an effective strategy. Based on our findings
and observations, we provide some recommendations for instructors and students
of classes at this level.
Study at a glance
- Question: How well can different versions of ChatGPT solve college-level engineering-math and electromagnetism problems?
- Setting and data: 13 college-level engineering-math and electromagnetism problems typically given to sophomore electrical engineering majors.
- Method: Tests and compares how multiple versions of ChatGPT solve problems across repeated trials.
Why it matters
Understanding how ChatGPT 3.5, 4, and 4 with Code Interpreter perform on college-level engineering-math and electromagnetism problems informs instructors and students working with sophomore electrical engineering coursework.
Key findings
- On sophomore college-level engineering-math and electromagnetism problems, ChatGPT-4 with Code Interpreter satisfactorily solved most problems most of the time.
- ChatGPT-4 with Code Interpreter showed a major performance improvement over ChatGPT-4 and 3.5 without the tool.
- Because ChatGPT's performance was inconsistent, generating multiple attempts in new chats and selecting the most common answer was an effective strategy.
Summary drafted with AI from the paper’s abstract and may contain errors. See the full paper for details.
What is the application?
Who age?
Why use AI?
Study design
