Date
Publisher
arXiv
The growing emphasis on 21st-century competencies in postsecondary education, intensified by the transformative impact of generative AI, underscores the need to evaluate how these competencies are embedded in curricula and how effectively academic programs align with evolving workforce and societal demands. Curricular Analytics, particularly recent generative AI-powered approaches, offer a promising data-driven pathway. However, analyzing 21st-century competencies requires pedagogical reasoning beyond surface-level information retrieval, and the capabilities of large language models in this context remain underexplored. In this study, we extend prior curricular analytics research by examining a broader range of curriculum documents, competency frameworks, and models. Using 7,600 manually annotated curriculum-competency alignment scores, we assess the informativeness of different curriculum sources, benchmark general-purpose LLMs for curriculum-to-competency mapping, and analyze error patterns. We further introduce a reasoning-based prompting strategy, Curricular CoT, to strengthen LLMs' pedagogical reasoning. Our results show that detailed instructional activity descriptions are the most informative type of curriculum document for competency analytics. Open-weight LLMs achieve accuracy comparable to proprietary models on coarse-grained tasks, demonstrating their scalability and cost-effectiveness for institutional use. However, no model reaches human-level precision in fine-grained pedagogical reasoning. Our proposed Curricular CoT yields modest improvements by reducing bias in instructional keyword inference and improving the detection of nuanced pedagogical evidence in long text. Together, these findings highlight the untapped potential of institutional curriculum documents and provide an empirical foundation for advancing AI-driven curricular analytics.
Study at a glance
- Question: How accurately can large language models evaluate 21st-century competencies in college course documents, and can reasoning prompts help?
- Setting and data: 200 postsecondary courses from Open Syllabus, two US college catalogs, and Canvas records from a US university, covering 7,600 annotated course-competency pairs.
- Method: Benchmark evaluation and regression analysis comparing model ratings to human annotations across four classification levels.
Why it matters
Evaluating how postsecondary curricula foster 21st-century competencies is labor-intensive and hard to scale, while the ability of large language models to perform the necessary pedagogical reasoning across varied curriculum documents remains underexplored.
Key findings
- While all evaluated language models achieved over 70% accuracy on coarse binary classification, no model matched human precision in fine-grained five-class pedagogical reasoning.
- Detailed learning activity descriptions from learning management systems were the most informative curriculum documents, whereas instructional schedules had insufficient information up to 45% of the time.
- A structured reasoning strategy, Curricular CoT, modestly improved accuracy and reduced models' tendency to overestimate competency coverage, especially for larger models.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
What is the application?
Who is the user?
Who age?
Why use AI?
Study design
