Chat Debugging: An Exploratory Study Of Human-AI Collaboration To Debug Analog Circuits

This research paper describes an exploratory study on the effectiveness of Chat Debugging: troubleshooting malfunctioning analog circuits on breadboards and printed circuit boards (PCB) by undergraduates through conversations with public-domain large language models (LLMs). Through thematic analysis of students' voluntarily shared chat logs when debugging pre-determined buggy circuits under exam and time pressure, we discovered multimodal usage patterns by students and considerable domain knowledge and sensible debugging suggestions offered by off-the-shelf LLMs. Meanwhile, we also identified major gaps in LLM technologies and students' skills during human-AI collaborative debugging, such as LLMs' limitations in 2D/3D image-based reasoning, unjustified tone of confidence, and students' deficits in fundamental concepts and critical thinking.