Date
Publisher
arXiv
Virtual Teaching Assistants (VTAs) powered by Large Language Models (LLMs)
have the potential to enhance student learning by providing instant feedback
and facilitating multi-turn interactions. However, empirical studies on their
effectiveness and acceptance in real-world classrooms are limited, leaving
their practical impact uncertain. In this study, we develop an LLM-based VTA
and deploy it in an introductory AI programming course with 477 graduate
students. To assess how student perceptions of the VTA's performance evolve
over time, we conduct three rounds of comprehensive surveys at different stages
of the course. Additionally, we analyze 3,869 student--VTA interaction pairs to
identify common question types and engagement patterns. We then compare these
interactions with traditional student--human instructor interactions to
evaluate the VTA's role in the learning process. Through a large-scale
empirical study and interaction analysis, we assess the feasibility of
deploying VTAs in real-world classrooms and identify key challenges for broader
adoption. Finally, we release the source code of our VTA system, fostering
future advancements in AI-driven education:
\texttt{https://github.com/sean0042/VTA}.
Study at a glance
- Question: How do students perceive and interact with an AI virtual teaching assistant in a real-world classroom?
- Setting and data: Introductory AI programming course with 477 graduate students; 3,869 student-AI teaching assistant interaction pairs.
- Method: Three student surveys across the course, analysis of student-AI interactions, and comparison with traditional student-instructor interactions.
Why it matters
Virtual Teaching Assistants powered by large language models have the potential to enhance student learning through instant feedback and multi-turn interactions. However, limited empirical research on their classroom effectiveness and acceptance leaves their practical impact uncertain.
Key findings
- In an introductory AI programming course with 477 graduate students, deploying an AI virtual teaching assistant was feasible, though key adoption challenges remain.
- AI virtual teaching assistants have the potential to enhance student learning by providing instant feedback and facilitating multi-turn interactions.
- Students generated 3,869 interactions with the AI assistant, revealing common question types and engagement patterns compared to interactions with human instructors.
Summary drafted with AI from the paper’s abstract and may contain errors. See the full paper for details.
What is the application?
Who is the user?
Who age?
Why use AI?
Study design
