Exploring The Role Of Tracing In AI-Supported Planning For Algorithmic Reasoning

Yoshee Jain, University of Illinois Urbana-Champaign

Heejin Do, ETH Zurich

Zihan Wu, University of Maine

Date

02/2026

AI Overview

Why it matters

Natural-language algorithm plans are often ambiguous, making them difficult for learners to translate into code and for AI models to interpret accurately when providing formative feedback before implementation.

Study at a glance

Question: How does manually tracing an algorithm plan affect students' planning, coding, and AI-generated feedback?
Setting and data: 20 college students from a large public university in the United States completing a remote programming task.
Method: Exploratory between-subjects lab experiment with mixed methods (qualitative plan analysis, mixed-effects regression, surveys, and interviews).
AI tools: GPT-4o

Key findings

Students using plan tracing wrote plans with fewer steps and control-flow references, focusing on higher-level goals and rationale rather than concrete code-like operations.
Plan tracing did not significantly alter the accuracy or explanation quality of AI-generated feedback, with error localization correct in 74% of tracing submissions versus 80% without tracing.
Coding performance was comparable between groups, but students in the plan-tracing condition reported lower confidence in their ability to solve similar problems in the future.

Summary drafted with AI from the full paper and may contain errors. See the full paper for details.

AI-powered planning tools show promise in supporting programming learners by enabling early, formative feedback on their thinking processes prior to coding. To date, however, most AI-supported planning tools rely on students' natural-language explanations, using LLMs to interpret learners' descriptions of their algorithmic intent. Prior to the emergence of LLM-based systems, CS education research extensively studied trace-based planning in pen-and-paper settings, demonstrating that reasoning through stepwise execution with explicit state transitions helps learners build and refine mental models of program behavior. Despite its potential, little is known about how tracing interacts with AI-mediated feedback and whether integrating tracing into AI-supported planning tools leads to different learning processes or interaction dynamics compared to natural-language-based planning alone. We study how requiring learners to produce explicit execution traces with an AI-supported planning tool affects their algorithmic reasoning. In a between-subjects study with 20 students, tracing shifted learners away from code-like, line-by-line descriptions toward more goal-driven reasoning about program behavior. Moreover, it led to more consistent partially correct solutions, although final coding performance remained comparable across conditions. Notably, tracing did not significantly affect the quality or reliability of LLM-generated feedback. These findings reveal tradeoffs in combining tracing with AI-supported planning and inform design guidelines for integrating natural language, tracing, and coding to support iterative reasoning throughout the programming process.