Date
Publisher
arXiv
Feedback is a critical aspect of improvement. Unfortunately, when there is a
lot of feedback from multiple sources, it can be difficult to distill the
information into actionable insights. Consider student evaluations of teaching
(SETs), which are important sources of feedback for educators. They can give
instructors insights into what worked during a semester. A collection of SETs
can also be useful to administrators as signals for courses or entire programs.
However, on a large scale as in high-enrollment courses or administrative
records over several years, the volume of SETs can render them difficult to
analyze. In this paper, we discuss a novel method for analyzing SETs using
natural language processing (NLP) and large language models (LLMs). We
demonstrate the method by applying it to a corpus of 5,000 SETs from a large
public university. We show that the method can be used to extract, embed,
cluster, and summarize the SETs to identify the themes they express. More
generally, this work illustrates how to use the combination of NLP techniques
and LLMs to generate a codebook for SETs. We conclude by discussing the
implications of this method for analyzing SETs and other types of student
writing in teaching and research settings.
Study at a glance
- Question: How can artificial intelligence be used to organize and identify themes from large numbers of student evaluations?
- Setting and data: 5,000 student evaluations of teaching from a large public university.
- Method: Demonstration of a new text analysis method to extract, cluster, and summarize feedback (natural language processing and large language models).
Why it matters
Student evaluations of teaching provide important feedback for educators and administrators. However, large volumes of feedback are difficult to distill into actionable insights and analyze at scale.
Key findings
- Across 5,000 student evaluations of teaching at a large public university, language models successfully created a qualitative codebook of feedback themes.
- Generative AI can extract and group written student comments to identify the main themes learners express.
- Language models can organize large volumes of course evaluations that are typically difficult for educators and administrators to analyze.
Summary drafted with AI from the paper’s abstract and may contain errors. See the full paper for details.
What is the application?
Who is the user?
Who age?
Why use AI?
Study design
