Date
Publisher
arXiv
With the rapid advancement of LLM models, they have become widely useful in various fields. While these AI systems can be used for code generation, significantly simplifying and accelerating the tasks of developers, their use for students to do assignments has raised ethical questions in the field of education. In this context, determining the author of a particular code becomes important. In this study, we introduce AIGCodeSet, a dataset for AI-generated code detection tasks, specifically for the Python programming language. We obtain the problem descriptions and human-written codes from the CodeNet dataset. Using the problem descriptions, we generate AI-written codes with CodeLlama 34B, Codestral 22B, and Gemini 1.5 Flash models in three approaches: i) generating code from the problem description alone, ii) generating code using the description along with human-written source code containing runtime errors, and iii) generating code using the problem description and human-written code that resulted in wrong answers. Lastly, we conducted a post-processing step to eliminate LLM output irrelevant to code snippets. Overall, AIGCodeSet consists of 2,828 AI-generated and 4,755 human-written code snippets. We share our code with the research community to support studies on this important topic and provide performance results for baseline AI-generated code detection methods.
Study at a glance
- Question: How accurately can machine learning baseline methods detect AI-generated Python code across different models and usage scenarios?
- Setting and data: 7,583 Python code samples (4,755 human-written from IBM CodeNet and 2,828 AI-generated across 317 programming problems).
- Method: Machine learning benchmark evaluation comparing Random Forest, XGBoost, Support Vector Machines, and a Bayesian classifier.
Why it matters
Distinguishing AI-generated code from human-written code is critical for software security and ethical standards, such as addressing students having AI models do their homework and candidates using AI tools in job interviews.
Key findings
- A Bayesian classifier achieved the highest overall detection sensitivity and F1 score of 0.63, while an SVM with Ada embeddings reached 0.71 accuracy.
- Detectors performed best on code generated from scratch (up to 0.985 recall for the Bayes classifier) but struggled far more on AI-fixed code.
- Code generated by Gemini 1.5 Flash was the most difficult to detect, yielding lower recall scores than Codestral 22B or CodeLlama 34B.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
What is the application?
Who age?
Why use AI?
Study design
