Search and Filter

Measuring Teaching With Llms

Date

AI Overview

Why it matters

Expert human ratings of teaching quality are expensive, unreliable, and have unknown accuracy, while general-purpose large language models struggle to reliably evaluate authentic classroom observations.

Study at a glance
  • Question: Can specialized language models reliably assess classroom teaching quality across established observation frameworks?
  • Setting and data: Transcripts of 1,600 elementary math lessons from 317 fourth- and fifth-grade teachers across four U.S. school districts.
  • Method: Model evaluation using multilevel partial rank correlations, variance decomposition, and canonical correlation analysis.
  • AI tools: RoBERTa, Unsupervised SimCSE, Supervised SimCSE, E5, Multilingual E5, GTE
Key findings
  • Three sentence-embedding models achieved human-level performance, with correlations exceeding 0.65 for general teaching practices and surpassing average human rater correlations for math instruction.
  • As models matured during training, they attributed more score variation to whole-lesson features rather than isolated spoken turns.
  • Aggregate model scores aligned with teacher effects on student learning gains, but this relationship did not hold for individual rubric items.

Summary drafted with AI from the full paper and may contain errors. See the full paper for details.

Objective and scalable measurement of teaching quality is a persistent challenge in education. While Large Language Models (LLMs) offer potential, general-purpose models have struggled to reliably apply complex, authentic classroom observation instruments. This paper uses custom LLMs built on sentence-level embeddings, an architecture better suited for the long-form, interpretive nature of classroom transcripts than conventional subword tokenization. We systematically evaluate five different sentence embeddings under a data-efficient training regime designed to prevent overfitting. Our results demonstrate that these specialized models can achieve human-level and even super-human performance with expert human ratings above 0.65 and surpassing the average human-human rater correlation. Further, through analysis of annotation context windows, we find that more advanced models-those better aligned with human judgments-attribute a larger share of score variation to lesson-level features rather than isolated utterances, challenging the sufficiency of single-turn annotation paradigms. Finally, to assess external validity, we find that aggregate model scores align with teacher value-added measures, indicating they are capturing features relevant to student learning. However, this trend does not hold at the individual item level, suggesting that while the models learn useful signals, they have not yet achieved full generalization. This work establishes a viable and powerful new methodology for AI-driven instructional measurement, offering a path toward providing scalable, reliable, and valid feedback for educator development.