Date
Publisher
arXiv
AI programming tutors provide scalable support, yet lack the behavioral context human tutors rely on to adapt support to learners' needs. We present TutorTrace, a dataset and behavioral abstraction pipeline that makes learners' behavioral context visible and computable in real time from low-level IDE telemetry. Across four deployments in two introductory Python courses (N=480), TutorTrace captures approximately 180K telemetry events, 13,633 behavioral segments, and 27 continuously computed metrics. From this foundation, we derive a taxonomy of learner activity before the first AI query, between consecutive queries, and across the full session, enabling systems to respond not just to what learners say, but to what they have done leading up to the help-seeking moment. In a preliminary classroom evaluation, behavior-aware prompts were associated with a decrease in intervals between queries with no independent work from 50.0% to 20.7%. As an additional demonstration of downstream utility, we evaluate TutorTrace on two held-out prediction tasks: whether a learner will query within the next 60 seconds (AUROC=.726) and whether an upcoming query reflects guided or dependent help-seeking (AUROC=.717). Together, these findings show how behavioral context can enable adaptive AI tutoring at scale.
Study at a glance
- Question: Can real-time behavioral tracking help AI tutors recognize student struggle and adapt their pedagogical support?
- Setting and data: Two introductory Python courses at Virginia Tech across four deployments, including 480 students.
- Method: System development, observational clustering, nonrandomized between-cohort classroom comparison, and predictive machine learning models.
Why it matters
AI programming tutors provide scalable support but lack the behavioral context that human tutors rely on to adapt help. Current systems typically respond only to learners' explicit questions rather than the actions that preceded them.
Key findings
- Behavior-aware AI prompting was associated with a decrease in passive windows between queries from 50.0% to 20.7% in a classroom deployment.
- First queries from students who made no prior coding attempt were 96.7% dependent, compared to around 30% for students who attempted code first.
- Using observable behavioral metrics to predict upcoming student help-seeking improved classification accuracy to an AUROC of 0.726 for query imminence and 0.717 for help-seeking type.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
What is the application?
Who is the user?
Who age?
Why use AI?
Study design
