Date
AI Overview
Why it matters
Institution-level educational functions like course evaluation remain heavily reliant on manual analysis, making large-scale reviews time-consuming, difficult to scale, and vulnerable to inconsistency.
Study at a glance
- Question: How closely do AI-generated course evaluations align with expert judgments and provide interpretable feedback?
- Setting and data: 100 authentic university courses from a recent institutional review at a major university in China.
- Method: Case study comparing AI and expert ratings using correlation, Bland–Altman, intraclass correlation, and qualitative feedback analysis.
- AI tools: Kimi, Llama, Llama-UKP
Key findings
- A domain-adapted model, Llama-UKP, achieved substantial agreement with expert judgments, reaching a Pearson correlation of 0.754 and a Spearman correlation of 0.843.
- The general-purpose model Kimi showed a strong high-score tendency, rating 73% of courses as excellent compared to 27% under Llama-UKP and 18% under human experts.
- Llama-UKP demonstrated consistent relative ordering and absolute agreement with experts, with most score differences falling within the 95% limits of agreement.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
The rapid adoption of generative AI has created new opportunities for teaching, learning, and quality assurance. Existing applications, however, remain largely student-facing, with comparatively limited attention to institution-level needs. This paper presents a course intelligence platform deployed across more than 100 universities and serving over 10,000 instructors in China. By linking competency requirements, knowledge structures, teaching activities, and assessment evidence, it establishes a shared foundation for knowledge organization, instructional design, learning assessment, and quality evaluation. The course evaluation module is examined as a representative institution-facing application of the platform, which integrates national evaluation standards, structured educational evidence, customized prompting strategies, and domain-adapted LLMs to generate quantitative scores and qualitative feedback. A case study involving 100 authentic university courses is conducted to evaluate its alignment with expert judgments and the interpretability of its outputs. Statistical analyses show substantial agreement between AI-generated assessments and expert ratings, while qualitative results highlight the credibility of the feedback. The findings further suggest that AI-assisted evaluation requires not only capable models but also structured domain knowledge and transparent criteria. In this context, human ratings should be treated as an informative reference rather than an error-free gold standard, and the objective is to achieve consistent, interpretable, and defensible judgments instead of merely replicating expert scores.
Research Categories
What is the application?
Who age?
Why use AI?
AI tools
Publication Source
Publisher
arXiv
