Repository Feedback

Your feedback helps us improve the repository's content relevance and usability. Please share your thoughts to help us better serve researchers and practitioners.

Submit feedback

Submit a research study

Contribute to the repository:

Add a paper

"Give me the code" - Log Analysis of First-Year CS Students' Interactions With GPT

Authors
Pedro Alves,
Bruno Pereira Cipriano
Date
Publisher
arXiv
Link
http://arxiv.org/pdf/2411.17855v2
The impact of Large Language Models (LLMs) like GPT-3, GPT-4, and Bard in computer science (CS) education is expected to be profound. Students now have the power to generate code solutions for a wide array of programming assignments. For first-year students, this may be particularly problematic since the foundational skills are still in development and an over-reliance on generative AI tools can hinder their ability to grasp essential programming concepts. This paper analyzes the prompts used by 69 freshmen undergraduate students to solve a certain programming problem within a project assignment, without giving them prior prompt training. We also present the rules of the exercise that motivated the prompts, designed to foster critical thinking skills during the interaction. Despite using unsophisticated prompting techniques, our findings suggest that the majority of students successfully leveraged GPT, incorporating the suggested solutions into their projects. Additionally, half of the students demonstrated the ability to exercise judgment in selecting from multiple GPT-generated solutions, showcasing the development of their critical thinking skills in evaluating AI-generated code.
Study at a glance
  • Question: How do first-year computer science students use GPT to solve a programming assignment without prior prompt training?
  • Setting and data: Prompts from 69 freshmen undergraduate students working on a project programming problem.
  • Method: Log analysis of student prompts
Why it matters

Students can now generate code solutions for many assignments. For first-year students still developing foundational skills, an over-reliance on generative AI tools can hinder their ability to grasp essential programming concepts.

Key findings
  • In a college computer science course with 69 freshmen, findings suggest the majority successfully incorporated GPT-generated code into their project assignments.
  • Half of the students demonstrated critical thinking by exercising judgment when selecting from multiple GPT-generated solutions.
  • Students relied on unsophisticated prompting techniques when attempting to solve programming problems without prior training.

Summary drafted with AI from the paper’s abstract and may contain errors. See the full paper for details.

What is the application?
Learning – Student Support
Who is the user?
Student
Who age?
Post-Secondary
Why use AI?
Outcomes – Durable Skills
Study design
Descriptive – Implementation and Use,
Quantitative – Others
Download PDF