Date
AI Overview
Why it matters
School policies often permit minor AI assistance in writing, but existing AI detectors assume a simple human-versus-machine binary, ignoring realistic student collaboration and creating risks of false accusations.
Study at a glance
- Question: How accurately do AI text detectors identify student writing across realistic levels of AI assistance?
- Setting and data: 916 human-written English essays from IELTS test-takers, U.S. 6th–12th graders, and UK university students, plus 12,703 AI-generated variants.
- Method: Benchmark evaluation testing four detection methods across eight levels of human-AI collaboration and institutional policy boundaries.
- AI tools: GPT-4o-mini, Llama-3.3-70b, DIPPER, T5, Ghostbuster, Fast-DetectGPT
Key findings
- Detector performance drops sharply on texts with intermediate assistance, particularly AI-assisted rewrites of human-written essays.
- When institutional policies allow AI-assisted revisions, true positive rates drop toward zero under strict false-positive constraints.
- Detection accuracy is lower for texts generated by GPT-4o-mini compared to Llama-3.3-70b, across different corpora, and for shorter essays.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
Students increasingly use the assistance of large language models (LLMs) in their academic writing. While slight assistance (e.g., grammar and style correction, as well as feedback) is permitted under most institutional policies, it is usually forbidden to offload entire writing tasks to LLMs. Unfortunately, current approaches to LLM-generated text detection predominantly assume a binary distinction between human-written and LLM-generated text, ignoring the breadth of realistic human-AI collaboration practices and limiting the validity of detection systems for educational assessment. In this paper, we propose a contribution-aware evaluation framework for LLM-based detection systems in education. We introduce a scale of eight student contribution levels that model realistic writing scenarios ranging from fully human-written texts to LLM-assisted revisions to fully LLM-generated and adversarially humanized texts. Institutional policies regarding LLM use can then be translated to thresholds of acceptable LLM assistance on this scale. We further present Generative Essay Detection in Education (GEDE), a novel benchmark dataset comprising more than 900 human-written and over 12,500 generated essays across 886 tasks and all contribution levels. Using this benchmark, we conduct a systematic evaluation of four state-of-the-art zero-shot and supervised detection methods across policy boundaries, contribution levels, generative models, out-of-distribution data, and text length. We show that most detectors struggle to accurately classify texts at intermediate student contribution levels, in particular LLM-assisted revisions of human-written texts. Such errors pose a substantial risk of false accusations, indicating that current text detection systems are (still) unsuitable to reliably support the enforcement of institutional policies regarding LLM assistance in education.
Research Categories
What is the application?
Who is the user?
Who age?
Why use AI?
Study design
Methods
AI tools
Publication Source
Publisher
arXiv
