Date
Publisher
arXiv
Purpose: Higher education institutions face increasing pressure to audit course designs for generative AI (GenAI) integration. This paper presents an end-to-end method for using large language models (LLMs) to scan course information sheets at scale, identify where assessments may be vulnerable to student use of GenAI tools, validate system performance through iterative refinement, and operationalise results through direct stakeholder communication and effort.
Method: We developed a four-phase pipeline: (0) manual pilot sampling, (1) iterative prompt engineering with multi-model comparison, (2) full production scan of 4,684 Bachelor and Master course information sheets (Academic Year 2024-2025) from the Vrije Universiteit Brussel (VUB) with automated report generation and email distribution to teaching teams (91.4% address-matched) using a three-tier risk taxonomy (Clear risk, Potential risk, Low risk), and (3) longitudinal re-scan of 4,675 sheets after the next catalogue release.
Results: Five iterations of prompt refinement achieved 87% agreement with expert labels. GPT-4o was selected for production based on superior handling of ambiguous cases involving internships and practical components. The Year 1 scan classified 60.3% of courses as Clear risk, 15.2% as Potential risk, and 24.5% as Low risk. Year 2 comparison revealed substantial shifts in risk distributions, with improvements most pronounced in practice-oriented programmes.
Implications: The method enables institutions to rapidly transform heterogeneous catalogue data into structured and actionable intelligence. The approach is transferable to other audit domains (sustainability, accessibility, pedagogical alignment) and provides a template for responsible LLM deployment in higher education governance.
Study at a glance
- Question: Can large language models reliably audit university course information sheets at scale to identify assessment risks from generative AI?
- Setting and data: Vrije Universiteit Brussel, Belgium; 4,684 course sheets in 2024–2025 and 4,675 in 2025–2026 across Bachelor and Master programs.
- Method: Iterative prompt development on a 30-course pilot, followed by automated LLM document classification and longitudinal descriptive comparison.
Why it matters
Higher education institutions face urgent pressure to audit whether course evaluations remain valid as generative AI tools emerge, but manually reviewing thousands of heterogeneous course documents is too labor-intensive.
Key findings
- In the Year 1 scan of 4,684 courses, 60.3% were classified as Clear risk, 15.2% as Potential risk, and 24.5% as Low risk.
- By Year 2, the Clear risk share dropped to 48.2%, with the largest improvements occurring in practice-oriented programs such as physical education, physiotherapy, and urban planning.
- Five iterative prompt refinement cycles achieved 87% agreement with expert rater labels on the pilot sample, with GPT-4o outperforming other tested models on ambiguous edge cases.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
What is the application?
Who is the user?
Who age?
Why use AI?
Study design
