Date
Publisher
arXiv
Effective feedback is essential for fostering students' success in scientific inquiry. With advancements in artificial intelligence, large language models (LLMs) offer new possibilities for delivering instant and adaptive feedback. However, this feedback often lacks the pedagogical validation provided by real-world practitioners. To address this limitation, our study evaluates and compares the feedback quality of LLM agents with that of human teachers and science education experts on student-written experimentation protocols. Four blinded raters, all professionals in scientific inquiry and science education, evaluated the feedback texts generated by 1) the LLM agent, 2) the teachers and 3) the science education experts using a five-point Likert scale based on six criteria of effective feedback: Feed Up, Feed Back, Feed Forward, Constructive Tone, Linguistic Clarity, and Technical Terminology. Our results indicate that LLM-generated feedback shows no significant difference to that of teachers and experts in overall quality. However, the LLM agent's performance lags in the Feed Back dimension, which involves identifying and explaining errors within the student's work context. Qualitative analysis highlighted the LLM agent's limitations in contextual understanding and in the clear communication of specific errors. Our findings suggest that combining LLM-generated feedback with human expertise can enhance educational practices by leveraging the efficiency of LLMs and the nuanced understanding of educators.
Study at a glance
- Question: How does the quality of AI-generated feedback on student science work compare to feedback from teachers and experts?
- Setting and data: Student-written experimentation protocols and feedback texts from teachers, science education experts, and an AI agent.
- Method: Blinded evaluation comparing AI, teacher, and expert feedback across six quality criteria using rating scales.
Why it matters
Effective feedback is essential for students' success in scientific inquiry. Although large language models offer new possibilities for delivering instant and adaptive feedback, this feedback often lacks pedagogical validation provided by real-world practitioners.
Key findings
- On student-written science experimentation protocols, AI feedback showed no significant difference in overall quality compared to feedback from teachers and science education experts.
- AI feedback lagged behind human educators in identifying and explaining errors within the context of a student's work.
- The AI tool showed limitations in contextual understanding and in clearly communicating specific errors.
Summary drafted with AI from the paper’s abstract and may contain errors. See the full paper for details.
What is the application?
Who is the user?
Who age?
Why use AI?
