Generative AI offers new opportunities for individualized and adaptive learning, e.g., through large language model (LLM)-based feedback systems. While LLMs can produce effective feedback for relatively straightforward conceptual tasks, delivering high-quality feedback for tasks that require advanced domain expertise, such as physics problem solving, remains a substantial challenge. This study presents the design of an LLM-based feedback system for physics problem solving grounded in evidence-centered design (ECD) and evaluates its performance within the German Physics Olympiad. Participants assessed the usefulness and accuracy of the generated feedback, which was generally perceived as useful and highly accurate. However, an in-depth analysis revealed that the feedback contained errors in 20% of cases; errors that often went unnoticed by the students. We discuss the risks associated with uncritical reliance on LLM-based feedback and outline potential directions for generating more adaptive and reliable LLM-based feedback in the future.
- Question: How useful and correct is large language model-generated feedback on students' physics problem-solving approaches?
- Setting and data: German Physics Olympiad participants, 38 secondary school students across Germany, 64 ratings and 47 written elaborations across six problems.
- Method: System development and pilot evaluation using student ratings, open-text responses, and expert error analysis (Mann-Whitney U test).
Delivering high-quality feedback for complex tasks like physics problem solving remains a challenge. It is unclear whether large language models can provide useful, factually correct feedback on multi-step reasoning rather than just simple conceptual tasks.
- German Physics Olympiad participants generally perceived the automated feedback as useful, with an average rating of 3.6 out of 5, and highly correct, averaging 4.4 out of 5.
- Expert evaluation revealed that the generated feedback contained physical or mathematical errors in about 20% of cases (13 of 64).
- Students rarely detected these errors, showing no significant difference in perceived correctness between correct and incorrect feedback, though some criticized the system for limited adaptivity to alternative solutions.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
