Date
Publisher
arXiv
Massive pre-trained language models have garnered attention and controversy
due to their ability to generate human-like responses: attention due to their
frequent indistinguishability from human-generated phraseology and narratives,
and controversy due to the fact that their convincingly presented arguments and
facts are frequently simply false. Just how human-like are these responses when
it comes to dialogues about physics, in particular about the standard content
of introductory physics courses? This study explores that question by having
ChatGTP, the pre-eminent language model in 2023, work through representative
assessment content of an actual calculus-based physics course and grading the
responses in the same way human responses would be graded. As it turns out,
ChatGPT would narrowly pass this course while exhibiting many of the
preconceptions and errors of a beginning learner.
Study at a glance
- Question: Can an artificial intelligence tool pass an introductory calculus-based physics course?
- Setting and data: Representative assessment content from an actual calculus-based physics course completed by ChatGPT in 2023.
- Method: Having an AI model complete physics course assessments and grading its responses the same way human work is graded.
Why it matters
Language models generate convincingly human-like responses, but their arguments and facts are frequently simply false. It remains unclear how human-like these responses are when addressing the standard content of introductory physics courses.
Key findings
- In an introductory, calculus-based physics course, ChatGPT generated responses that would narrowly earn a passing grade when graded like human students.
- ChatGPT exhibited many of the same preconceptions and conceptual errors typical of a beginning learner.
- The AI tool presented convincing, human-like phrasing and arguments that were frequently factually false.
Summary drafted with AI from the paper’s abstract and may contain errors. See the full paper for details.
What is the application?
Who is the user?
Who age?
Why use AI?
Study design
