Date
Publisher
arXiv
Generative AI and large language models hold great promise in enhancing
programming education by automatically generating individualized feedback for
students. We investigate the role of generative AI models in providing human
tutor-style programming hints to help students resolve errors in their buggy
programs. Recent works have benchmarked state-of-the-art models for various
feedback generation scenarios; however, their overall quality is still inferior
to human tutors and not yet ready for real-world deployment. In this paper, we
seek to push the limits of generative AI models toward providing high-quality
programming hints and develop a novel technique, GPT4Hints-GPT3.5Val. As a
first step, our technique leverages GPT-4 as a ``tutor'' model to generate
hints -- it boosts the generative quality by using symbolic information of
failing test cases and fixes in prompts. As a next step, our technique
leverages GPT-3.5, a weaker model, as a ``student'' model to further validate
the hint quality -- it performs an automatic quality validation by simulating
the potential utility of providing this feedback. We show the efficacy of our
technique via extensive evaluation using three real-world datasets of Python
programs covering a variety of concepts ranging from basic algorithms to
regular expressions and data analysis using pandas library.
Study at a glance
- Question: Can AI models provide helpful, human-style hints to help students fix errors in their computer programs?
- Setting and data: Three real-world datasets of Python programs covering basic algorithms, regular expressions, and data analysis using the pandas library.
- Method: Tests an AI technique that generates programming hints and evaluates their quality using a simulated student model.
Why it matters
Generative AI holds promise for providing individualized feedback to help students resolve programming errors. However, the quality of automated hints remains inferior to human tutors and is not yet ready for real-world deployment.
Key findings
- Across three datasets of Python programming, an AI system pairing hint generation with simulated student validation effectively produced high-quality hints for fixing errors.
- Existing generative AI models produce programming feedback that remains inferior to human tutors and is not yet ready for real-world deployment.
- Providing details about failing tests and code fixes directly to the AI model boosted the quality of its generated programming hints.
Summary drafted with AI from the paper’s abstract and may contain errors. See the full paper for details.
What is the application?
Who is the user?
Who age?
Why use AI?
Study design
