Date
Publisher
arXiv
Asynchronous learning environments (ALEs) are widely adopted for formal and
informal learning, but timely and personalized support is often limited. In
this context, Virtual Teaching Assistants (VTAs) can potentially reduce the
workload of instructors, but rigorous and pedagogically sound evaluation is
essential. Existing assessments often rely on surface-level metrics and lack
sufficient grounding in educational theories, making it difficult to
meaningfully compare the pedagogical effectiveness of different VTA systems. To
bridge this gap, we propose an evaluation framework rooted in learning sciences
and tailored to asynchronous forum discussions, a common VTA deployment context
in ALE. We construct classifiers using expert annotations of VTA responses on a
diverse set of forum posts. We evaluate the effectiveness of our classifiers,
identifying approaches that improve accuracy as well as challenges that hinder
generalization. Our work establishes a foundation for theory-driven evaluation
of VTA systems, paving the way for more pedagogically effective AI in
education.
Study at a glance
- Question: How effectively can educational frameworks and AI models evaluate the pedagogical quality of virtual teaching assistant responses?
- Setting and data: 300 university forum questions from 116 students across 85 courses, plus 104 academic questions from the Stanford MOOC dataset.
- Method: Framework development, expert annotation, and model benchmark evaluation comparing zero-shot prompting, prompt optimization, and supervised fine-tuning.
Why it matters
Virtual teaching assistants can provide scalable support in asynchronous online learning, but existing assessments rely on surface-level metrics and lack educational theory grounding to evaluate pedagogical effectiveness.
Key findings
- Including peer forum context reduced simulated virtual teaching assistant performance, decreasing disciplinary understanding scores by 1 or 2 points in 59.7% of responses.
- Automated classifiers performed reliably at lower pedagogical levels such as clarifying misunderstandings, but accuracy declined substantially at higher levels like metacognitive awareness.
- Fine-tuning on synthetic data improved classifier F-1 scores by up to 41.7%, though classifiers showed limited generalization and sensitivity to markdown formatting across datasets.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
What is the application?
Who is the user?
Who age?
Why use AI?
Study design
