Date
AI Overview
Why it matters
Students frequently seek help from AI tutors during programming, but little is known about which response styles best support continued learning across different student situations and confusion levels.
Study at a glance
- Question: Which AI tutoring response styles are most associated with productive student engagement across different programming help-seeking situations?
- Setting and data: Undergraduate artificial intelligence course, UMass Amherst, Fall 2024 and Spring 2025; 16,851 interactions from 2,214 dialogues across 203 students.
- Method: Descriptive and chi-square analysis of dialogue turns annotated using a local language model.
- AI tools: ChatGPT, Gemma 4
Key findings
- Across all interactions, response styles showed statistically significant but small differences, with verification feedback yielding the highest productive continuation rate (82.4%) and direct answers the lowest (62.7%).
- Response effectiveness varied most widely in high-cognitive-load and debugging contexts, but showed very little variation in low-confusion conceptual questions.
- Direct answers were associated with higher unresolved continuation during complex debugging, whereas stepwise guidance was associated with greater confusion reduction during code requests.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
As students increasingly use LLM tutors in computer science education, one question becomes especially important: what kind of response helps a student continue productively? Prior work has studied how students use LLMs in computer science education, but less is known about how tutoring response styles are associated with student follow-up across programming help-seeking contexts. This paper analyzes StudyChat (UMass, 2026), a public dataset of student and ChatGPT tutoring conversations from an artificial intelligence course. We transformed StudyChat into 16,851 assistant-response interactions from 203 students and 2,214 conversations. Using local LLM-assisted annotation with Gemma 4, we labeled student help-seeking situations, student state, assistant response style, and student next-turn outcome. Human validation showed 82\% agreement with the LLM-assisted labels (Cohen's $κ=.74$). We analyzed productive continuation and unresolved continuation across the full dataset and across help-seeking contexts. Globally, response style was significantly associated with productive continuation, $χ^2(7)=100.39$, $p<.001$, $V=.078$, and unresolved continuation, $χ^2(7)=125.77$, $p<.001$, $V=.087$, though effect sizes were small. Verification feedback had the highest productive-continuation rate (82.4\%), while direct answers had the lowest (62.7\%). Descriptively, response-style score ranges were smallest in low-confusion conceptual contexts (.017) and largest in high-cognitive-load contexts (.203). More detailed comparisons showed situation-dependent response patterns. For example, stepwise guidance was followed by greater confusion decrease in high-cognitive-load code requests, while direct answers were followed by more unresolved continuation in high-load debugging. These findings support context-aware evaluation and design of AI tutoring responses for programming education.
Research Categories
What is the application?
Who is the user?
Who age?
Why use AI?
Study design
AI tools
Datasets
Publication Source
Publisher
arXiv
