Search and Filter

Designing for Novice Debuggers: A Pilot Study on an Al-Assisted Debugging Tool

Date

AI Overview

Why it matters

Many AI debugging tools provide complete solutions to semantic errors rather than guiding learners step by step, risking over-reliance and hindering independent problem-solving skills.

Study at a glance
  • Question: How usable and effective is an interactive AI debugging assistant designed to guide novice programmers through fixing semantic errors?
  • Setting and data: 10 first- and second-year undergraduate students at the Singapore University of Technology and Design.
  • Method: Usability testing comparing two debugging tools in a counterbalanced sequence, using surveys, System Usability Scale, and system interaction logs.
  • AI tools: CodeHinter, GPT-4o, ChatGPT, SID
Key findings
  • Undergraduate participants gave the new tool an average System Usability Scale score of 75/100 and a satisfaction rating of 89/100, outperforming the previous version.
  • All participants used the 'End-to-End Test' feature, and 'Locate Lines With Errors' was the most frequently accessed and highly favored helper tool.
  • Six out of ten students used the 'Provide Code Solution' button, sometimes multiple times to check and compare their answers.

Summary drafted with AI from the full paper and may contain errors. See the full paper for details.

Debugging is a fundamental skill that novice programmers must develop. Numerous tools have been created to assist novice programmers in this process. Recently, large language models (LLMs) have been integrated with automated program repair techniques to generate fixes for students' buggy code. However, many of these tools foster an over-reliance on AI and do not actively engage students in the debugging process. In this work, we aim to design an intuitive debugging assistant, CodeHinter, that combines traditional debugging tools with LLM-based techniques to help novice debuggers fix semantic errors while promoting active engagement in the debugging process. We present findings from our second design iteration, which we tested with a group of undergraduate students. Our results indicate that the students found the tool highly effective in resolving semantic errors and significantly easier to use than the first version. Consistent with our previous study, error localization was the most valuable feature. Finally, we conclude that any AI-assisted debugging tool should be personalized based on user profiles to optimize their interactions with students.