Date
Publisher
arXiv
Students' explanations of scientific phenomena provide important evidence of their conceptual understanding. However, because conceptual understanding can only be inferred through language rather than observed directly, distinguishing conceptual understanding from linguistic quality represents a fundamental challenge for assessment. This challenge can also be expected to arise when artificial intelligence (AI) is used to score students' text-based explanations. We investigated whether AI-based scoring approaches assess students' conceptual understanding independently of the linguistic quality of their explanations about a phenomenon taken from a physics context. Conceptual understanding scores assigned by nine machine learning-based scoring approaches and two large language model-based scoring approaches were compared with expert-assigned scores for 116 explanations produced by secondary-school students in Germany. Multinomial logistic regression analyses examined whether expert-rated linguistic quality was associated with under- or overestimation of conceptual understanding. Across all eleven AI-based scoring approaches, lower linguistic quality was associated with a greater likelihood of underestimation, although the strength and statistical significance of this association varied between approaches. AI-based scoring approaches may thus systematically underestimate the conceptual understanding expressed in linguistically weaker explanations. This pattern resembles language bias previously documented in STEM teachers' assessment practices. The findings highlight the importance of examining not only agreement with expert ratings but also whether AI-based scoring approaches inadvertently rely on construct-irrelevant information. Addressing language bias is therefore essential for the validity and fairness of AI-supported assessment in physics education and STEM education more broadly.
Study at a glance
- Question: Does the linguistic quality of students' science explanations influence how artificial intelligence scores their conceptual understanding?
- Setting and data: Seven Grade 9 classes across four secondary schools in Hamburg, Germany; 116 authentic written physics explanations.
- Method: Multinomial logistic regression comparing AI-assigned and expert-assigned scores, with repeated sampling for large language models.
Why it matters
Distinguishing students' conceptual understanding from their linguistic quality is a fundamental assessment challenge in STEM. If artificial intelligence scoring is influenced by writing quality, language bias could operate at scale and disadvantage students.
Key findings
- Across all 11 AI scoring approaches, lower writing quality was associated with 1.3- to 3.9-times higher odds of underestimating students' conceptual understanding in physics.
- Higher linguistic quality was generally unrelated to overestimating conceptual understanding, with positive associations appearing in only two machine-learning setups.
- Agreement between AI-assigned and expert-assigned conceptual scores ranged from 67.2% to 78.4% across all tested approaches.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
What is the application?
Who is the user?
Who age?
Why use AI?
Study design
