Date
Publisher
arXiv
Artificial Intelligence (AI), with ChatGPT as a prominent example, has recently taken center stage in various domains including higher education, particularly in Computer Science and Engineering (CSE). The AI revolution brings both convenience and controversy, offering substantial benefits while lacking formal guidance on their application. The primary objective of this work is to comprehensively analyze the pedagogical potential of ChatGPT in CSE education, understanding its strengths and limitations from the perspectives of educators and learners. We employ a systematic approach, creating a diverse range of educational practice problems within CSE field, focusing on various subjects such as data science, programming, AI, machine learning, networks, and more. According to our examinations, certain question types, like conceptual knowledge queries, typically do not pose significant challenges to ChatGPT, and thus, are excluded from our analysis. Alternatively, we focus our efforts on developing more in-depth and personalized questions and project-based tasks. These questions are presented to ChatGPT, followed by interactions to assess its effectiveness in delivering complete and meaningful responses. To this end, we propose a comprehensive five-factor reliability analysis framework to evaluate the responses. This assessment aims to identify when ChatGPT excels and when it faces challenges. Our study concludes with a correlation analysis, delving into the relationships among subjects, task types, and limiting factors. This analysis offers valuable insights to enhance ChatGPT's utility in CSE education, providing guidance to educators and students regarding its reliability and efficacy.
Study at a glance
- Question: What are the strengths and limitations of ChatGPT for teaching and learning computer science and engineering?
- Setting and data: Practice problems and project tasks across computer science and engineering subjects in higher education.
- Method: Tests ChatGPT on custom computer science tasks, evaluating answers using a five-factor reliability framework and correlation analysis.
Why it matters
The AI revolution brings both convenience and controversy to computer science education, offering substantial benefits while lacking formal guidance on its application. Educators and learners need to understand ChatGPT's pedagogical potential, strengths, and limitations.
Key findings
- In college computer science and engineering, conceptual knowledge questions typically do not pose significant challenges for ChatGPT.
- ChatGPT faces challenges when tasked with delivering complete and meaningful responses to in-depth, personalized, and project-based problems.
- The factors limiting ChatGPT's reliability are linked to the specific subject area and the type of learning task assigned.
Summary drafted with AI from the paper’s abstract and may contain errors. See the full paper for details.
What is the application?
Who age?
Why use AI?
Study design
