Date
Publisher
arXiv
The potential of artificial intelligence (AI)-based large language models
(LLMs) holds considerable promise in revolutionizing education, research, and
practice. However, distinguishing between human-written and AI-generated text
has become a significant task. This paper presents a comparative study,
introducing a novel dataset of human-written and LLM-generated texts in
different genres: essays, stories, poetry, and Python code. We employ several
machine learning models to classify the texts. Results demonstrate the efficacy
of these models in discerning between human and AI-generated text, despite the
dataset's limited sample size. However, the task becomes more challenging when
classifying GPT-generated text, particularly in story writing. The results
indicate that the models exhibit superior performance in binary classification
tasks, such as distinguishing human-generated text from a specific LLM,
compared to the more complex multiclass tasks that involve discerning among
human-generated and multiple LLMs. Our findings provide insightful implications
for AI text detection while our dataset paves the way for future research in
this evolving area.
Study at a glance
- Question: Can computer models accurately tell the difference between human-written and AI-generated text across different writing genres?
- Setting and data: Dataset of human-written and AI-generated texts, including essays, stories, poetry, and Python code.
- Method: Comparative study testing multiple machine learning models to classify texts.
Why it matters
Artificial intelligence-based large language models hold considerable promise for revolutionizing education, research, and practice. However, distinguishing between human-written and AI-generated text has become a significant task.
Key findings
- Across essays, stories, poetry, and computer code, detection models effectively distinguished between human-written and artificial intelligence-generated text.
- Detecting AI-generated text was more challenging for GPT-produced content, especially in creative story writing.
- Detection models were more accurate at distinguishing human writing from one specific AI tool than identifying texts across multiple different AI models.
Summary drafted with AI from the paper’s abstract and may contain errors. See the full paper for details.
What is the application?
Who is the user?
Who age?
Why use AI?
Study design
