Date
Publisher
arXiv
Generative AI offers new opportunities for individualized and adaptive learning, e.g., through large language model (LLM)-based feedback systems. While LLMs can produce factually correct feedback for relatively straightforward conceptual tasks, delivering high-quality feedback for tasks that require advanced domain expertise, such as physics problem solving, remains a substantial challenge. This study presents the design and implementation of an LLM-based feedback system for physics problem solving grounded in evidence-centered design and reports a first evaluation within the German Physics Olympiad. Participants rated the usefulness and correctness of the generated feedback for each implemented problem. The collected ratings indicate that the feedback was generally perceived as useful and highly correct. However, an in-depth analysis revealed that the feedback contained errors in 20% of cases; errors that often went unnoticed by the students. We discuss the risks associated with uncritical reliance on LLM-based feedback and outline potential directions for generating more adaptive and reliable LLM-based feedback in the future.
Study at a glance
- Question: How useful and correct is evidence-centered AI feedback for students solving complex physics problems?
- Setting and data: 38 secondary school students participating in the first stage of the German Physics Olympiad, evaluating 64 feedback instances across six problems.
- Method: System development and user evaluation using surveys, open-text comments, and expert error analysis with Mann-Whitney U testing.
Why it matters
Developing physics problem solving requires deliberate practice with timely feedback, but human feedback is resource-limited. Whether large language models can provide useful, factually correct feedback on complex multi-step physics problem solving remains largely unexamined.
Key findings
- Expert evaluation revealed that the AI-generated feedback contained errors in about 20% of cases, ranging from calculation errors to misclassifying correct alternative solutions.
- Students generally perceived the AI feedback as highly correct and useful, with average ratings of 4.4 and 3.6 out of 5, respectively.
- Students rarely detected errors in the feedback, showing no statistically significant difference in perceived correctness between correct and incorrect AI responses.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
What is the application?
Who is the user?
Who age?
Why use AI?
Study design
