Automating Human Tutor-Style Programming Feedback: Leveraging GPT-4 Tutor Model for Hint Generation and GPT-3.5 Student Model for Hint Validation

Tung Phung, Max Planck Institute for Software Systems

Victor-Alexandru Padurean, Max Planck Institute for Software Systems

Anjali Singh, University of Michigan

Christopher Brooks, University of Michigan

Jose Cambronero, Microsoft (United States)

Sumit Gulwani, Microsoft (United States)

Adish Singla, Max Planck Institute for Software Systems

Gustavo Soares, Microsoft (United States)

Date

08/2024

AI Overview

Why it matters

Large class enrollments make it infeasible for human tutors to promptly provide individualized programming feedback. While AI models can generate feedback, their quality remains inferior to human tutors and unready for classroom deployment.

Study at a glance

Question: Can combining symbolic prompt data with simulated student validation improve AI-generated programming hints to human tutor levels?
Setting and data: 79 buggy Python programs across three datasets from online course students and GeeksforGeeks users.
Method: Model evaluation comparing AI hint generation against baseline models and human tutors using expert rubric ratings.
AI tools: GPT-4 (gpt-4-0613), ChatGPT (gpt-3.5-turbo-0613), GPT4Hints-GPT3.5Val

Key findings

The GPT4Hints-GPT3.5Val method achieved around 95% precision, matching human tutor quality while maintaining over 70% coverage across three real-world Python datasets.
Using GPT-3.5 as a simulated student validated hint quality better than GPT-4, as the stronger model often fixed bugs even with incorrect explanations.
Prompting GPT-4 with symbolic information like failing test cases and fixed programs improved baseline hint precision from 66.0-85.4% to 82.0-93.3% without validation.

Summary drafted with AI from the full paper and may contain errors. See the full paper for details.

Generative AI and large language models hold great promise in enhancing programming education by automatically generating individualized feedback for students. We investigate the role of generative AI models in providing human tutor-style programming hints to help students resolve errors in their buggy programs. Recent works have benchmarked state-of-the-art models for various feedback generation scenarios; however, their overall quality is still inferior to human tutors and not yet ready for real-world deployment. In this paper, we seek to push the limits of generative AI models toward providing high-quality programming hints and develop a novel technique, GPT4Hints-GPT3.5Val. As a first step, our technique leverages GPT-4 as a ``tutor'' model to generate hints -- it boosts the generative quality by using symbolic information of failing test cases and fixes in prompts. As a next step, our technique leverages GPT-3.5, a weaker model, as a ``student'' model to further validate the hint quality -- it performs an automatic quality validation by simulating the potential utility of providing this feedback. We show the efficacy of our technique via extensive evaluation using three real-world datasets of Python programs covering a variety of concepts ranging from basic algorithms to regular expressions and data analysis using pandas library.