Date
AI Overview
Why it matters
The widespread availability of chatbots offers rich personalized tutoring opportunities, but educators fear student over-reliance or misuse. Understanding real-world interaction behavior is necessary to inform effective tutoring systems and better models of student learning.
Study at a glance
- Question: How do university students interact with an AI chatbot during programming assignments, and how does usage relate to course outcomes?
- Setting and data: University of Massachusetts Amherst AI course, Fall 2024 and Spring 2025, 203 undergraduate students, 2,214 conversations, and 924 programming submissions.
- Method: Observational study using automated dialogue act classification, linear regression, and K-means clustering.
- AI tools: StudyChat, ChatGPT, gpt-4o-mini, GPT4.1
Key findings
- Asking conceptual questions or seeking coding help was positively associated with assignment and exam performance, while contextual follow-up questions were negatively correlated with assignment scores.
- Students asking the chatbot to write assignment reports averaged lower exam scores (83.8%) than those asking targeted coding questions (89.0%).
- High-frequency chatbot users achieved higher minimum exam and assignment scores and less grade variance than low- or medium-frequency users, despite similar overall averages.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
The widespread availability of large language models (LLMs), such as ChatGPT, has significantly impacted education, raising both opportunities and challenges. Students can frequently interact with LLM-powered, interactive learning tools, but their usage patterns need to be observed and understood. We introduce StudyChat, a publicly available dataset capturing real-world student interactions with an LLM-powered tutoring chatbot in a semester-long, university-level artificial intelligence (AI) course. We deploy a web application that replicates ChatGPT's core functionalities, and use it to log student interactions with the LLM while working on programming assignments. We collect 16,851 interactions, which we annotate using a dialogue act labeling schema inspired by observed interaction patterns and prior research. We analyze these interactions, highlight usage trends, and analyze how specific student behavior correlates with their course outcome. We find that students who prompt LLMs for conceptual understanding and coding help tend to perform better on assignments and exams. Moreover, students who use LLMs to write reports and circumvent assignment learning objectives have lower outcomes on exams than others. StudyChat serves as a shared resource to facilitate further research on the evolving role of LLMs in education.
Research Categories
What is the application?
Who is the user?
Who age?
Why use AI?
Publication Source
Publisher
arXiv
