Search and Filter

Repository Feedback

Your feedback helps us improve the repository's content relevance and usability. Please share your thoughts to help us better serve researchers and practitioners.

Submit feedback

Submit a research study

Contribute to the repository:

Add a paper

Topic Discovery And Classification For Responsible Generative Ai Adaptation In Higher Education

Date
Publisher
arXiv
As generative artificial intelligence (GenAI) becomes increasingly capable of delivering personalized learning experiences and real-time feedback, a growing number of students are incorporating these tools into their academic workflows. They use GenAI to clarify concepts, solve complex problems, and, in some cases, complete assignments by copying and pasting model-generated contents. While GenAI has the potential to enhance learning experience, it also raises concerns around misinformation, hallucinated outputs, and its potential to undermine critical thinking and problem-solving skills. In response, many universities, colleges, departments, and instructors have begun to develop and adopt policies to guide responsible integration of GenAI into learning environments. However, these policies vary widely across institutions and contexts, and their evolving nature often leaves students uncertain about expectations and best practices. To address this challenge, the authors designed and implemented an automated system for discovering and categorizing AI-related policies found in course syllabi and institutional policy websites. The system combines unsupervised topic modeling techniques to identify key policy themes with large language models (LLMs) to classify the level of GenAI allowance and other requirements in policy texts. The developed application achieved a coherence score of 0.73 for topic discovery. In addition, GPT-4.0-based classification of policy categories achieved precision between 0.92 and 0.97, and recall between 0.85 and 0.97 across eight identified topics. By providing structured and interpretable policy information, this tool promotes the safe, equitable, and pedagogically aligned use of GenAI technologies in education. Furthermore, the system can be integrated into educational technology platforms to help students understand and comply with relevant guidelines.
Study at a glance
  • Question: Can automated machine learning and language models identify and classify generative AI policy themes from educational texts?
  • Setting and data: 212 segmented policy documents and syllabi from 22 accredited post-secondary institutions across the United States.
  • Method: Unsupervised topic modeling (BERTopic) evaluated with coherence scores, followed by large language model classification benchmarked against faculty-labeled ground truth.
Why it matters

Generative AI policies vary widely across institutions and evolve rapidly, leaving students confused. Establishing clear guidelines demands significant resources and expertise, posing substantial challenges for under-resourced institutions.

Key findings
  • Unsupervised topic modeling paired with OpenAI Small embeddings identified eight key policy themes, achieving the highest coherence score of 0.73.
  • GPT-4.0 outperformed other tested models in classifying policy statements, achieving precision between 0.92 and 0.97 and recall between 0.85 and 0.97 across categories.
  • Expert evaluation of university policies revealed eight recurring categories, including permitted use in learning, assignments, assessments, research, citation rules, accuracy validation, and data cautions.

Summary drafted with AI from the full paper and may contain errors. See the full paper for details.

What is the application?
Who is the user?
Who age?
Why use AI?