Date
Publisher
arXiv
This research paper describes an exploratory study on the effectiveness of Chat Debugging: troubleshooting malfunctioning analog circuits on breadboards and printed circuit boards (PCB) by undergraduates through conversations with public-domain large language models (LLMs). Through thematic analysis of students' voluntarily shared chat logs when debugging pre-determined buggy circuits under exam and time pressure, we discovered multimodal usage patterns by students and considerable domain knowledge and sensible debugging suggestions offered by off-the-shelf LLMs. Meanwhile, we also identified major gaps in LLM technologies and students' skills during human-AI collaborative debugging, such as LLMs' limitations in 2D/3D image-based reasoning, unjustified tone of confidence, and students' deficits in fundamental concepts and critical thinking.
Why it matters
Little is known about the effectiveness of conversational large language models in helping undergraduates troubleshoot malfunctioning analog circuits on breadboards and printed circuit boards under exam and time pressure.
Key findings
- Off-the-shelf chatbots provided undergraduates with substantial domain knowledge and sensible troubleshooting suggestions while debugging malfunctioning analog circuits under exam pressure.
- AI tools showed major limitations in 2D and 3D image-based reasoning alongside an unjustified tone of confidence.
- Students demonstrated deficits in fundamental concepts and critical thinking during collaborative troubleshooting with AI models.
Summary drafted with AI from the paper’s abstract and may contain errors. See the full paper for details.
What is the application?
Who is the user?
Who age?
Why use AI?
Study design
