Date
Publisher
arXiv
Recent studies have integrated large language models (LLMs) into diverse
educational contexts, including providing adaptive programming hints, a type of
feedback focuses on helping students move forward during problem-solving.
However, most existing LLM-based hint systems are limited to one single hint
type. To investigate whether and how different levels of hints can support
students' problem-solving and learning, we conducted a think-aloud study with
12 novices using the LLM Hint Factory, a system providing four levels of hints
from general natural language guidance to concrete code assistance, varying in
format and granularity. We discovered that high-level natural language hints
alone can be helpless or even misleading, especially when addressing next-step
or syntax-related help requests. Adding lower-level hints, like code examples
with in-line comments, can better support students. The findings open up future
work on customizing help responses from content, format, and granularity levels
to accurately identify and meet students' learning needs.
Study at a glance
- Question: How do different levels of AI-generated programming hints support or hinder novice students during problem-solving?
- Setting and data: 12 undergraduate and graduate non-CS majors from two US research universities (one public, one private).
- Method: Think-aloud user study with pre- and post-tests, log analysis, and structured interviews.
Why it matters
Novice programmers need timely feedback when stuck, but most AI hint systems offer only one hint type. Evidence is limited on how different hint levels affect student learning and problem-solving.
Key findings
- Worked-example code hints most frequently supported success, leading to correct code modifications in 59.32% of instances.
- High-level natural language hints alone were often insufficient for next-step logic and syntax requests, causing frustration in 33.3% of students.
- High-level natural language hints were sufficient for debugging logic errors, but all misleading hints that caused errors were high-level.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
What is the application?
Who is the user?
Who age?
Why use AI?
Study design
