Search and Filter

You're (Not) My Type - Can LLMs Generate Feedback of Specific Types for Introductory Programming Tasks?

Date

AI Overview

Why it matters

Automated feedback systems for novice programmers traditionally offer limited, simple feedback. Large language models could provide richer support, but whether their output can be reliably steered into specific, pedagogically sound feedback types remains unproven.

Study at a glance
  • Question: Can prompts guide large language models to generate specific, controlled types of feedback for introductory programming tasks?
  • Setting and data: 11 authentic student programming submissions across 6 tasks in Java, Python, C++, and Kotlin from prior research datasets, evaluated in March 2024.
  • Method: Iterative prompt design evaluated across 66 generated feedback messages using expert consensual qualitative coding.
  • AI tools: GPT-4, ChatGPT
Key findings
  • In 63 out of 66 cases, the model successfully generated the requested feedback type, though unrequested feedback types were also included in 19 instances.
  • The model frequently combined hints on how to proceed or task constraints with explanations pointing out specific code errors.
  • Misleading feedback still occurred, including incorrectly labeling style issues as mistakes, misinterpreting input formatting requirements, and miscalculating progress percentages for correct solutions.

Summary drafted with AI from the full paper and may contain errors. See the full paper for details.

Background: Feedback as one of the most influential factors for learning has been subject to a great body of research. It plays a key role in the development of educational technology systems and is traditionally rooted in deterministic feedback defined by experts and their experience. However, with the rise of generative AI and especially Large Language Models (LLMs), we expect feedback as part of learning systems to transform, especially for the context of programming. In the past, it was challenging to automate feedback for learners of programming. LLMs may create new possibilities to provide richer, and more individual feedback than ever before. Objectives: This paper aims to generate specific types of feedback for introductory programming tasks using LLMs. We revisit existing feedback taxonomies to capture the specifics of the generated feedback, such as randomness, uncertainty, and degrees of variation. Methods: We iteratively designed prompts for the generation of specific feedback types (as part of existing feedback taxonomies) in response to authentic student programs. We then evaluated the generated output and determined to what extent it reflected certain feedback types. Results and Conclusion: The present work provides a better understanding of different feedback dimensions and characteristics. The results have implications for future feedback research with regard to, for example, feedback effects and learners' informational needs. It further provides a basis for the development of new tools and learning systems for novice programmers including feedback generated by AI.