Date
AI Overview
Why it matters
Universities are exploring alternatives to in-person examinations, raising concerns about academic integrity and pedagogical alignment in uninvigilated settings where generative AI is accessible. Evidence is needed to understand how generative AI performs holistically across university mathematics curricula.
Study at a glance
- Question: How well does generative AI perform across an entire standard first-year undergraduate mathematics curriculum compared to students?
- Setting and data: Eight first-year mathematics core course examinations, UK Russell Group university, 2024–2025 academic year, 128 students and 165 AI question responses.
- Method: Blind-marking transcribed AI exam responses alongside student scripts, combined with statistical resampling and direct estimation.
- AI tools: ChatGPT, chatgpt-4o-latest
Key findings
- Generative AI achieved an overall average score of 71.1 out of 100, performing on par with the student cohort average of 69.1.
- Across the entire curriculum, generative AI scores were markedly more consistent than student scores, showing substantially less variation.
- Generative AI performance varied across subjects, reaching 86.2 in Multivariable Calculus but dropping to 53.8 in Integration after assuming material beyond the course scope.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
Generative artificial intelligence (GenAI) tools such as OpenAI's ChatGPT are transforming the educational landscape, prompting reconsideration of traditional assessment practices. In parallel, universities are exploring alternatives to in-person, closed-book examinations, raising concerns about academic integrity and pedagogical alignment in uninvigilated settings. This study systematically investigates the performance of GenAI on typical mathematics questions from across a first-year mathematics curriculum. Adopting an empirical approach and utilising current examination questions as a proxy for course content, we generate, transcribe, and blind-mark GenAI submissions to eight undergraduate mathematics assessments, spanning the entirety of the first-year curriculum. By combining independent GenAI responses to individual questions, we enable a meaningful evaluation of GenAI performance, both at the level of modules and across the first-year curriculum. We find that GenAI attainment is at the level of a first-class degree, though current performance can vary between modules. Further, we find that GenAI performance is remarkably consistent when viewed across the entire curriculum, significantly more so than that of students in invigilated examinations. Our findings evidence the pressing need for redesigning assessments in mathematics in the era of generative artificial intelligence.
Research Categories
Who is the user?
Who age?
Why use AI?
Study design
AI tools
Publication Source
Publisher
arXiv
