Date
AI Overview
Why it matters
Generative AI can personalize learning and solve physics problems, but it frequently struggles with calculation errors and inconsistent outputs. Educators need insight into how students interact with AI tools to support conceptual learning rather than replacing it.
Study at a glance
- Question: How do introductory physics students interact with ChatGPT during problem solving, and what are their perceptions of using AI?
- Setting and data: Undergraduate students in a calculus-based introductory physics course at a northeastern U.S. community college, Fall 2024 and Spring 2025 semesters, 49 students.
- Method: Mixed methods analysis comparing assessment scores before and after AI access, combined with qualitative analysis of prompt transcripts and reflections.
- AI tools: ChatGPT
Key findings
- Students who engaged with AI tools improved by an average of 0.43 points, whereas those who chose not to use AI saw an average decrease of 1.8 points.
- Students who provided detailed problem context and asked conceptual questions received more accurate AI assistance and achieved larger gains than students seeking procedural problem-solving steps.
- About 20 percent of students chose not to use ChatGPT, primarily citing confidence in their own understanding or unfamiliarity with the tool.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
With the rise of large language models such as ChatGPT, interest has grown in
understanding how these tools influence learning in STEM education, including
physics. This study explores how students use ChatGPT during a physics
problem-solving task embedded in a formal assessment. We analyzed patterns of
AI usage and their relationship to student performance. Findings indicate that
students who engaged with ChatGPT generally performed better than those who did
not. Particularly, students who provided more complete and contextual prompts
experienced greater benefits. Further, students who demonstrated overall
positive gains collectively asked more conceptual questions than those who
exhibited overall negative gains. However, the presence of incorrect
AI-generated responses also underscores the importance of critically evaluating
AI output. These results suggest that while AI can be a valuable aid in problem
solving, its effectiveness depends significantly on how students use it,
reinforcing the need to incorporate structured AI-literacy into STEM education.
Research Categories
What is the application?
Who is the user?
Who age?
Why use AI?
Methods
AI tools
Publication Source
Publisher
arXiv
