Search and Filter

Repository Feedback

Your feedback helps us improve the repository's content relevance and usability. Please share your thoughts to help us better serve researchers and practitioners.

Submit feedback

Submit a research study

Contribute to the repository:

Add a paper

Students' Feedback Requests and Interactions with the SCRIPT Chatbot: Do They Get What They Ask For?

Authors
Andreas Scholl,
Natalie Kiesler
Date
Publisher
arXiv
Link
http://arxiv.org/pdf/2507.17258v1
Building on prior research on Generative AI (GenAI) and related tools for programming education, we developed SCRIPT, a chatbot based on ChatGPT-4o-mini, to support novice learners. SCRIPT allows for open-ended interactions and structured guidance through predefined prompts. We evaluated the tool via an experiment with 136 students from an introductory programming course at a large German university and analyzed how students interacted with SCRIPT while solving programming tasks with a focus on their feedback preferences. The results reveal that students' feedback requests seem to follow a specific sequence. Moreover, the chatbot responses aligned well with students' requested feedback types (in 75%), and it adhered to the system prompt constraints. These insights inform the design of GenAI-based learning support systems and highlight challenges in balancing guidance and flexibility in AI-assisted tools.
Study at a glance
  • Question: Does an AI coding chatbot provide students with the specific types of feedback they ask for?
  • Setting and data: 136 students in an introductory programming course at a large German university.
  • Method: Experiment analyzing student chatbot interactions during programming tasks.
Why it matters

Generative AI tools can support novice programming learners through open-ended interactions and structured guidance. However, designing these systems presents challenges in balancing guidance and flexibility while ensuring chatbots provide the specific feedback students ask for.

Key findings
  • In a German university programming course with 136 students, an AI chatbot's responses matched the feedback types learners requested 75% of the time.
  • Students' feedback requests seemed to follow a specific sequence while they solved programming tasks.
  • The chatbot adhered to its underlying system prompt constraints while interacting with students.

Summary drafted with AI from the paper’s abstract and may contain errors. See the full paper for details.

What is the application?
Learning – Student Support
Who is the user?
Student
Who age?
Post-Secondary
Why use AI?
Outcomes – Other Academic,
Outcomes – Differentiation,
Outcomes – Durable Skills
Study design
Descriptive – Implementation and Use,
Descriptive – Product Development,
Quantitative – Others
Download PDF