Search and Filter

Assessing GPTZero's Accuracy in Identifying AI vs. Human-Written Essays

Date

AI Overview

Why it matters

As students increasingly use generative AI, educators rely on AI detectors to prevent misuse. However, detectors risk wrongly identifying human work, meaning students who do not use AI could be wrongly punished.

Study at a glance
  • Question: Does essay length affect how accurately GPTZero detects AI-generated versus human-written text?
  • Setting and data: Fifty human student-written essays and 28 AI-generated essays.
  • Method: Benchmark evaluation comparing detector accuracy across three text length categories
  • AI tools: GPTZero, ChatGPT, ChatGPT 3.5, ChatGPT 4o
Key findings
  • GPTZero accurately classified AI-generated essays 100% of the time, assigning average AI probabilities between 97.00% and 99.17% across all lengths.
  • GPTZero misclassified 8 of 50 human-written essays as AI-generated, producing a 16% false positive rate and an overall error rate of 10.3%.
  • Human essays showed fluctuating AI probability scores across lengths, with the highest false-positive rate in short texts (35.56%) compared to medium (10.29%) and long texts (14.75%).

Summary drafted with AI from the full paper and may contain errors. See the full paper for details.

As the use of AI tools by students has become more prevalent, instructors have started using AI detection tools like GPTZero and QuillBot to detect AI written text. However, the reliability of these detectors remains uncertain. In our study, we focused mostly on the success rate of GPTZero, the most-used AI detector, in identifying AI-generated texts based on different lengths of randomly submitted essays: short (40-100 word count), medium (100-350 word count), and long (350-800 word count). We gathered a data set consisting of twenty-eight AI-generated papers and fifty human-written papers. With this randomized essay data, papers were individually plugged into GPTZero and measured for percentage of AI generation and confidence. A vast majority of the AI-generated papers were detected accurately (ranging from 91-100% AI believed generation), while the human generated essays fluctuated; there were a handful of false positives. These findings suggest that although GPTZero is effective at detecting purely AI-generated content, its reliability in distinguishing human-authored texts is limited. Educators should therefore exercise caution when relying solely on AI detection tools.

Research Categories

What is the application?
Who is the user?
Who age?
Why use AI?

Publication Source

Publisher
arXiv