Search and Filter

Repository Feedback

Your feedback helps us improve the repository's content relevance and usability. Please share your thoughts to help us better serve researchers and practitioners.

Submit feedback

Submit a research study

Contribute to the repository:

Add a paper

To Tab Or Not To Tab: Measuring Critical Engagement In AI Code Completion Tools Using Behavioral Signals And Attention Checks

Jessica Hutchison, Temple University
Ian Applebaum, Temple University
Kenneth Angelikas, Temple University
Kush Rakesh Patel, Temple University
Phuoc Nguyen, Temple University
Antonio Lazaro, Temple University
Nicholas Rucinski, Temple University
Rahad Arman Nabid, Temple University
Stephen MacNeil, Temple University
Date
Publisher
arXiv
Link
https://arxiv.org/abs/2606.30549v1
AI code completion tools, such as Github Copilot, provide students with code suggestions to help them write programs. However, recent qualitative studies suggest that students fail to critically evaluate these suggestions. We present Clover, a code completion tool that logs students' interactions with code suggestions and additionally offers attention checks to probe reflective engagement during programming tasks. We also develop a taxonomy of behavioral interaction metrics for AI-assisted programming, informed by literature. We analyzed relationships between interaction patterns, engagement with attention checks, and task performance. We observed that higher rates of tab accept were associated with lower attention check performance, while increased dwell time was associated with higher attention check performance. We conclude by discussing how programming process data and attention checks might support reflective engagement in AI-assisted programming.
Study at a glance
  • Question: How do students' interaction behaviors and attention check performance with an AI code assistant relate to programming task performance?
  • Setting and data: Temple University, 55 undergraduate students in a first-year Java programming course lab across four sections.
  • Method: Correlational analysis of student interaction logs and test case performance during an in-person lab task.
Why it matters

AI code completion tools can mislead novice programmers and disrupt metacognition, yet tracking how students interact with AI suggestions remains difficult to evaluate at scale.

Key findings
  • Students with higher rates of accepting code suggestions using the tab key were far more likely to fail deliberate attention checks.
  • Longer dwell times before acting on code suggestions were associated with better attention check performance.
  • Interaction behaviors such as code completion acceptance and dwell time were only weakly or inconsistently related to overall task performance.

Summary drafted with AI from the full paper and may contain errors. See the full paper for details.

What is the application?
Learning – Student Support
Who is the user?
Student
Who age?
Post-Secondary
Why use AI?
Outcomes – Other Academic,
Outcomes – Durable Skills
Study design
Descriptive – Product Development,
Quantitative – Others
Download PDF