Date
AI Overview
Why it matters
Unethical use of AI text generators threatens academic integrity, and existing detection models can be easily bypassed or lack evaluations on multi-authored documents.
Study at a glance
- Question: Can writing style analysis detect AI-generated text and identify author changes in collaborative documents?
- Setting and data: Benchmark human text dataset with four authors, combined with AI-generated text created under normal and strict prompts.
- Method: Machine learning evaluation of fine-tuned language models on four text classification and author identification tasks.
- AI tools: gemini-1.5-flash-latest, BERT, ALBERT, DistilBERT, RoBERTa
Key findings
- Fine-tuned models classified human versus machine text with high accuracy, reaching near-perfect F1-scores on normal prompts but dropping to around 0.94 on strict prompts.
- Distinguishing single-authored from multi-authored documents proved highly accurate, achieving F1-scores between 0.97 and 0.99 across both prompt types.
- Detecting paragraph-level author switches and identifying specific authors within multi-authored documents was much more challenging, with F1-scores dropping to around 0.50 to 0.70.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
This work addresses critical challenges to academic integrity, including plagiarism, fabrication, and verification of authorship of educational content, by proposing a Natural Language Processing (NLP)-based framework for authenticating students' content through author attribution and style change detection. Despite some initial efforts, several aspects of the topic are yet to be explored. In contrast to existing solutions, the paper provides a comprehensive analysis of the topic by targeting four relevant tasks, including (i) classification of human and machine text, (ii) differentiating in single and multi-authored documents, (iii) author change detection within multi-authored documents, and (iv) author recognition in collaboratively produced documents. The solutions proposed for the tasks are evaluated on two datasets generated with Gemini using two different prompts, including a normal and a strict set of instructions. During experiments, some reduction in the performance of the proposed solutions is observed on the dataset generated through the strict prompt, demonstrating the complexities involved in detecting machine-generated text with cleverly crafted prompts. The generated datasets, code, and other relevant materials are made publicly available on GitHub, which are expected to provide a baseline for future research in the domain.
Research Categories
What is the application?
Who is the user?
Who age?
Why use AI?
Study design
AI tools
Publication Source
Publisher
arXiv
