Date
Publisher
arXiv
As AI code tools become integrated into programming environments, students increasingly describe intended behavior in natural language and rely on these tools to generate code, shifting emphasis from code writing to specification. Yet little is known about the comments students write as specifications in AI-assisted programming tasks. We analyze a four-year dataset of undergraduate programming submissions and reflections from tasks in which students wrote comments to guide code generation and refined solutions using test-case feedback. We introduce a taxonomy spanning three dimensions: comment type, code expression level, and code construct. Using automated classification, we examine how these dimensions vary across attempts and how students describe the process in their reflections. Our findings show that students mostly wrote natural-language What comments, shifted toward How comments for more procedural constructs, and focused more on verifying generated code than on repeatedly rewriting comments.
Study at a glance
- Question: How do undergraduate students write and revise comments to guide an AI code generator, and what are their perspectives?
- Setting and data: Second-year undergraduate object-oriented programming course, 2023 to 2026, 1,161 students, 10,257 submission attempts, and 124,201 analyzed comment instances.
- Method: Mixed methods descriptive analysis and LLM-assisted content analysis of student submissions, comment revisions, and reflections.
Why it matters
As AI code-generation tools shift programming from writing code to specifying intent, little is known about how students write and revise comments to direct AI models.
Key findings
- Students mostly wrote natural-language comments describing intended behavior (77.0%), but shifted to procedural descriptions for multi-step blocks (94.3%) and loop iterations (57.3%).
- Across successive submission attempts, students rarely rewrote existing comments, with modifications accounting for only 4.4% of comments in the first ten attempts.
- Student reflections showed that the primary effort shifted from repeatedly rewriting prompts to reviewing, validating, and verifying AI-generated code.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
What is the application?
Who is the user?
Who age?
Study design
