Search and Filter

An Exploration Of Higher Education Course Evaluation By Large Language Models

Date

AI Overview

Why it matters

Higher education course evaluations are time-consuming, difficult to scale, and susceptible to inconsistency. Most educational AI remains student-facing, leaving institutional quality assurance and governance needs relatively unaddressed.

Study at a glance
  • Question: How closely do AI-generated course evaluations align with expert judgments in authentic higher education reviews?
  • Setting and data: A major university in China; course documentation and expert ratings from 100 authentic university courses.
  • Method: System development and case study comparing AI and expert ratings using correlation, Bland-Altman, and intraclass correlation analyses.
  • AI tools: Kimi, Llama, Llama-UKP
Key findings
  • A fine-tuned model (Llama-UKP) achieved strong correlation with human expert ratings (Pearson's r = 0.754, Spearman's rho = 0.843), outperforming a general-purpose model.
  • The general-purpose model rated 73% of courses as excellent (85 points or above), compared with 27% under the fine-tuned model and 18% under human experts.
  • The fine-tuned model demonstrated substantial absolute agreement with expert evaluations (intraclass correlation coefficient of 0.723) and produced evidence-grounded feedback linking scores to course materials.

Summary drafted with AI from the full paper and may contain errors. See the full paper for details.

Course evaluation plays a critical role in ensuring instructional quality and guiding curriculum development in higher education. However, traditional evaluation methods, such as student surveys, classroom observations, and expert reviews, are often constrained by subjectivity, high labor costs, and limited scalability. With recent advancements in large language models (LLMs), new opportunities have emerged for generating consistent, fine-grained, and scalable course evaluations. This study investigates the use of three representative LLMs for automated course evaluation at both the micro level (classroom discussion analysis) and the macro level (holistic course review). Using classroom interaction transcripts and a dataset of 100 courses from a major institution in China, we demonstrate that LLMs can extract key pedagogical features and generate structured evaluation results aligned with expert judgement. A fine-tuned version of Llama shows superior reliability, producing score distributions with greater differentiation and stronger correlation with human evaluators than its counterparts. The results highlight three major findings: (1) LLMs can reliably perform systematic and interpretable course evaluations at both the micro and macro levels; (2) fine-tuning and prompt engineering significantly enhance evaluation accuracy and consistency; and (3) LLM-generated feedback provides actionable insights for teaching improvement. These findings illustrate the promise of LLM-based evaluation as a practical tool for supporting quality assurance and educational decision-making in large-scale higher education settings.