Date
Publisher
arXiv
This is the study that presents an AI-Python-based chatbot that helps
students to learn programming by demonstrating solutions to such problems as
debugging errors, solving syntax problems or converting abstract theoretical
concepts to practical implementations. Traditional coding tools like Integrated
Development Environments (IDEs) and static analyzers do not give robotic help
while AI-driven code assistants such as GitHub Copilot focus on getting things
done. To close this gap, our chatbot combines static code analysis, dynamic
execution tracing, and large language models (LLMs) to provide the students
with relevant and practical advice, hence promoting the learning process. The
chatbots hybrid architecture employs CodeLlama for code embedding, GPT-4 for
natural language interactions, and Docker-based sandboxing for secure
execution. Evaluated through a mixed-methods approach involving 1,500 student
submissions, the system demonstrated an 85% error resolution success rate,
outperforming standalone tools like pylint (62%) and GPT-4 (73%). Quantitative
results revealed a 59.3% reduction in debugging time among users, with pre- and
post-test assessments showing a 34% improvement in coding proficiency,
particularly in recursion and exception handling. Qualitative feedback from 120
students highlighted the chatbots clarity, accessibility, and
confidence-building impact, though critiques included occasional latency and
restrictive code sanitization. By balancing technical innovation with
pedagogical empathy, this research provides a blueprint for AI tools that
prioritize educational equity and long-term skill retention over mere code
completion. The chatbot exemplifies how AI can augment human instruction,
fostering deeper conceptual understanding in programming education.
Study at a glance
- Question: Can an AI-powered pedagogical chatbot improve students' Python debugging efficiency, error resolution, and conceptual understanding?
- Setting and data: 120 to 150 students, a 4-week trial, and 1,500 student submissions from introductory Python courses.
- Method: Mixed methods combining design science research, a quasi-experimental pre- and post-test comparison, surveys, and system benchmark evaluations.
Why it matters
Novice programming students often struggle to understand cryptic error messages and debug complex code, while existing AI tools frequently supply direct code solutions without pedagogical explanations or safety safeguards.
Key findings
- The AI chatbot resolved 85% of errors across 1,500 student submissions, outperforming standalone GPT-4 (73%) and pylint (62%).
- Students using the chatbot reduced their average debugging time by 59.3%, compared to a 14% reduction among control students using traditional programming environments.
- Students using the chatbot achieved a 28.3% assessment gain score compared to 9.1% in the control group, with notable gains in recursion and exception handling.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
What is the application?
Who is the user?
Who age?
Why use AI?
