Date
Publisher
arXiv
Problem posing is a valuable learning activity in computing education, encouraging learners to actively construct, refine, and reflect on problems rather than simply solving them. This experience report presents the design and pilot deployment of two LLM-powered scaffolding systems for supporting problem posing across two computational scenarios with different levels of task openness. Both systems assessed student-generated problems using Bloom's Taxonomy-based criteria and applied the same assessment framework, differing only in output modality: one provided guiding questions (Indirect scaffolding), while the other offered worked examples (Direct scaffolding). We conducted a within-subjects, counterbalanced pilot study with 20 graduate students and collected problem-quality ratings, user-experience surveys, and post-session interviews. Our deployment showed that both systems supported problem refinement in complementary ways, each offering distinct benefits. Direct scaffolding produced greater immediate improvements, while interviews showed that participants valued Indirect scaffolding for promoting deeper reflection on their own problem design. Based on these findings, we suggest sequencing the two modalities by beginning with Indirect scaffolding to promote reflection, then shifting to Direct scaffolding when learners become stuck. These lessons offer an initial practical strategy for integrating LLM-based scaffolding into computing classrooms.
Study at a glance
- Question: How do direct worked examples versus indirect guiding questions from an AI shape students' problem-posing processes and quality?
- Setting and data: 20 computer science graduate students at a university, completing computational problem-posing tasks.
- Method: Counterbalanced within-subject user study with rubric-based problem ratings, user-experience surveys, and post-session interviews.
Why it matters
Instructors lack guidance on how AI scaffolding should support student thinking during open-ended tasks, risking cognitive offloading when AI assistance reduces opportunities for learners to critically evaluate their own work.
Key findings
- Both AI scaffolding approaches significantly improved problem quality, but direct scaffolding produced larger score gains than indirect scaffolding.
- Direct scaffolding received significantly higher pragmatic usability ratings, while interviews indicated participants valued indirect guiding questions for encouraging deeper reflection.
- Direct scaffolding generated significantly larger improvements than indirect scaffolding in an open-ended coin-change scenario, but neither approach produced significant advantages in a narrow handshake scenario.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
What is the application?
Who is the user?
Who age?
Why use AI?
Study design
