Search and Filter

Exploring The Potential Of ChatGPT For Feedback And Evaluation In Experimental Physics

Date

AI Overview

Why it matters

Evaluating physics lab reports is demanding, with persistent challenges regarding grading consistency and feedback reliability in large-enrollment courses. Integrating language models effectively and responsibly remains an open challenge.

Study at a glance
  • Question: How can an automated AI system assist in evaluating introductory physics lab reports, and what are its limitations?
  • Setting and data: 57 randomly selected undergraduate physics lab reports from 150 submissions, Universidad de la República, Uruguay, 2025 academic year.
  • Method: Mixed methods comparing AI and instructor grades descriptively and qualitatively coding AI-generated feedback across rubric criteria.
  • AI tools: GPT-5.4, ChatGPT
Key findings
  • AI-assigned scores showed weak rank agreement with instructor grades (Spearman's rank correlation of 0.38) and averaged lower than human scores (7.91 versus 8.63).
  • Across rubric criteria, most AI feedback correctly applied the rubric (81% to 89%), but 5% to 13% of responses were superficial.
  • Invalid AI evaluations occurred in up to 11% of responses, mainly driven by difficulties extracting or reading equations, figures, and graphs.

Summary drafted with AI from the full paper and may contain errors. See the full paper for details.

This study explores how generative artificial intelligence, specifically ChatGPT, can assist in the evaluation of laboratory reports in Experimental Physics. Two interaction modalities were implemented: an automated API-based evaluation and a customized ChatGPT configuration designed to emulate instructor feedback. The analysis focused on two complementary dimensions-formal and structural integrity, and technical accuracy and conceptual depth. Findings indicate that ChatGPT provides consistent feedback on organization, clarity, and adherence to scientific conventions, while its evaluation of technical reasoning and interpretation of experimental data remains less reliable. Each modality exhibited distinctive limitations, particularly in processing graphical and mathematical information. The study contributes to understanding how the use of AI in evaluating laboratory reports can inform feedback practices in experimental physics, highlighting the importance of teacher supervision to ensure the validity of physical reasoning and the accurate interpretation of experimental results.

Research Categories

What is the application?
Who is the user?
Who age?
Why use AI?

Publication Source

Publisher
arXiv