Date
AI Overview
Why it matters
As students increasingly use generative AI, educators rely on AI detectors to prevent misuse. However, detectors risk wrongly identifying human work, meaning students who do not use AI could be wrongly punished.
Study at a glance
- Question: Does essay length affect how accurately GPTZero detects AI-generated versus human-written text?
- Setting and data: Fifty human student-written essays and 28 AI-generated essays.
- Method: Benchmark evaluation comparing detector accuracy across three text length categories
- AI tools: GPTZero, ChatGPT, ChatGPT 3.5, ChatGPT 4o
Key findings
- GPTZero accurately classified AI-generated essays 100% of the time, assigning average AI probabilities between 97.00% and 99.17% across all lengths.
- GPTZero misclassified 8 of 50 human-written essays as AI-generated, producing a 16% false positive rate and an overall error rate of 10.3%.
- Human essays showed fluctuating AI probability scores across lengths, with the highest false-positive rate in short texts (35.56%) compared to medium (10.29%) and long texts (14.75%).
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
As the use of AI tools by students has become more prevalent, instructors
have started using AI detection tools like GPTZero and QuillBot to detect AI
written text. However, the reliability of these detectors remains uncertain. In
our study, we focused mostly on the success rate of GPTZero, the most-used AI
detector, in identifying AI-generated texts based on different lengths of
randomly submitted essays: short (40-100 word count), medium (100-350 word
count), and long (350-800 word count). We gathered a data set consisting of
twenty-eight AI-generated papers and fifty human-written papers. With this
randomized essay data, papers were individually plugged into GPTZero and
measured for percentage of AI generation and confidence. A vast majority of the
AI-generated papers were detected accurately (ranging from 91-100% AI believed
generation), while the human generated essays fluctuated; there were a handful
of false positives. These findings suggest that although GPTZero is effective
at detecting purely AI-generated content, its reliability in distinguishing
human-authored texts is limited. Educators should therefore exercise caution
when relying solely on AI detection tools.
Research Categories
What is the application?
Who is the user?
Who age?
Why use AI?
Study design
AI tools
Publication Source
Publisher
arXiv
