Date
Publisher
arXiv
PURPOSE OR GOAL: This study investigates how GenAI can be integrated with a criterion-referenced grading framework to improve the efficiency and quality of grading for mathematical assessments in engineering. It specifically explores the challenges demonstrators face with manual, model solution-based grading and how a GenAI-supported system can be designed to reliably identify student errors, provide high-quality feedback, and support human graders. The research also examines human graders' perceptions of the effectiveness of this GenAI-assisted approach. ACTUAL OR ANTICIPATED OUTCOMES: The study found that GenAI achieved an overall grading accuracy of 92.5%, comparable to two experienced human graders. The two researchers, who also served as subject demonstrators, perceived the GenAI as a helpful second reviewer that improved accuracy by catching small errors and provided more complete feedback than they could manually. A central outcome was the significant enhancement of formative feedback. However, they noted the GenAI tool is not yet reliable enough for autonomous use, especially with unconventional solutions. CONCLUSIONS/RECOMMENDATIONS/SUMMARY: This study demonstrates that GenAI, when paired with a structured, criterion-referenced framework using binary questions, can grade engineering mathematical assessments with an accuracy comparable to human experts. Its primary contribution is a novel methodological approach that embeds the generation of high-quality, scalable formative feedback directly into the assessment workflow. Future work should investigate student perceptions of GenAI grading and feedback.
Study at a glance
- Question: Can generative AI using binary questions accurately grade engineering mathematics assessments and support human graders?
- Setting and data: Undergraduate electrical engineering course at an Australian university; 50 student submissions across 15 pre-laboratory mathematical questions; reflections from two demonstrators.
- Method: Mixed methods combining qualitative reflections with a comparative grading accuracy and consistency evaluation.
Why it matters
Delivering timely, high-quality grading and formative feedback in large-scale mathematics assessments is time-intensive, while generative AI's potential to assist with grading engineering mathematics remains underexplored.
Key findings
- The AI system achieved an overall grading accuracy of 92.5% across 50 student submissions, closely matching the accuracy of experienced human instructors.
- Almost half of the AI's grading mistakes stemmed from interpretation differences, followed by simple oversights, unanticipated student approaches, and symbolic simplification misjudgments.
- Human graders valued the AI as a helpful second reviewer that caught minor errors and provided more complete feedback, though not yet reliable for autonomous grading.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
What is the application?
Who is the user?
Who age?
Why use AI?
Study design
