Search and Filter

Exploring The Value Of Diverse LLM Explanations In Introductory Programming

Date

AI Overview

Why it matters

Textbooks and static resources lack the flexibility to adapt to students' needs. While AI can immediately generate code explanations, it remains unclear whether providing multiple explanations with intentionally distinct conceptual emphases enhances novice programmers' understanding.

Study at a glance
  • Question: Do diverse AI-generated code explanations improve introductory programming students' understanding and perceptions more than generic explanations?
  • Setting and data: A 12-week first-year engineering programming course at a large public research university, fall 2024, 971 students.
  • Method: Randomized controlled trial (between-subjects experiment) combined with surveys and qualitative analysis.
  • AI tools: GPT-4o
Key findings
  • Open-ended question accuracy was consistently about 7.7% higher when students received diverse explanations rather than generic ones, though this difference was not statistically significant after adjustment.
  • Multiple-choice accuracy was consistently high across all conditions (exceeding 87%), with negligible and non-significant differences between diverse and generic explanation groups.
  • Students reported no significant differences in perceived explanation helpfulness or information overload, rating the overall information load similarly regardless of condition.

Summary drafted with AI from the full paper and may contain errors. See the full paper for details.

Large Language Models (LLMs) have shown the potential to generate code explanations that surpass those of peers in quality, offering promising opportunities for computer science education. While these explanations may not yet match the depth and clarity of instructor-provided explanations, research in computational creativity highlights that the quantity and diversity of ideas can often outweigh a singular focus on quality. Inspired by this, we explore whether combining multiple diverse explanations, each emphasizing distinct aspects (e.g., function, concept, goal), can enhance students' understanding of programming exercises compared to generic explanations that do not emphasize distinct conceptual aspects. In our study 971 first-year computing students were randomly assigned either diverse or generic LLM-generated explanations for two programming exercises. Students completed multiple-choice and open-ended questions for each exercise, followed by Likert-scale questions and open-ended reflections. Our findings outline patterns in student performance and perceived cognitive load across the two explanation conditions. These findings highlight how variation in explanation emphasis may relate to learner engagement and understanding. Across participants, open-ended response accuracy was consistently about 7.7% higher when students received diverse explanations, with no difference in perceived cognitive load.