Date
Publisher
arXiv
This study applies Computational Grounded Theory (CGT) to analyze student misconceptions using interaction data from an AI-powered chatbot deployed in a university-level Modern Physics course. The chatbot - the UTA Study Buddy Bot - engaged students in peer-like problem-solving conversations throughout the semester, generating a rich dataset of over 10 million tokens. To explore patterns in student reasoning and identify recurring conceptual difficulties, we implemented a CGT pipeline that combined natural language processing, unsupervised clustering of sentence-level vector embeddings, human interpretation of emergent themes, and supervised learning to evaluate the generalizability of identified categories. Preliminary results revealed persistent misconceptions in areas such as relativistic momentum and quantum energy levels, along with distinctive trends in how students phrased their questions and expressed uncertainty. These findings underscore the potential of CGT as a scalable, theory-aligned approach for extracting insights from chatbot dialogues and guiding the development of more adaptive, AI-driven educational tools in physics instruction.
Study at a glance
- Question: What physics misconceptions appear in student chatbot conversations, and how can computational methods help identify them?
- Setting and data: University of Texas at Arlington, Fall 2024, 47 consenting undergraduate modern physics students submitting 1,642 chatbot requests (1,504 analyzed messages).
- Method: Computational Grounded Theory using sentence embeddings, unsupervised clustering (BERTopic), human refinement, and supervised classification (logistic regression).
Why it matters
Qualitative physics education methods do not scale easily to large datasets. Analyzing student chatbot dialogues can capture authentic misconceptions and reasoning that traditional assessments miss.
Key findings
- Unsupervised clustering identified 47 fine-grained topics that aggregated into five macro-themes, led by energy, fusion, and forces, which accounted for roughly 65% of student messages.
- A logistic regression classifier predicted student macro-themes from sentence embeddings with 90% accuracy in 10-fold cross-validation.
- Chatbot conversations revealed persistent student misconceptions regarding relativistic kinetic energy versus rest mass energy, quantum state transitions, and harmonic oscillator calculations.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
What is the application?
Who is the user?
Who age?
Why use AI?
