On The Effectiveness Of Llm-Specific Fine-Tuning For Detecting Ai-Generated Text

Michał Gromadzki, Warsaw University of Technology

Anna Wróblewska, Warsaw University of Technology

Agnieszka Kaliska, Adam Mickiewicz University in Poznań

Date

01/2026

AI Overview

Why it matters

AI-generated text closely resembles human writing, challenging academic integrity and authenticity verification in education. Flawed detection tools often produce false positives or negatives, undermining confidence in detection and institutional policies.

Study at a glance

Question: Can fine-tuning language models on specific generators or model families accurately detect AI-generated text?
Setting and data: Benchmark of 10 human datasets and AI text from 21 language models, totaling 2.9 billion tokens.
Method: Model evaluation and benchmarking of fine-tuned language models and ensemble classifiers for token-level detection.
AI tools: gpt-4.1-nano-2025-04-14, Phi-4, Llama-3.1-8B-Instruct, Mistral-Nemo-Instruct-2407, Qwen2.5-14B-Instruct, Falcon3-7B-Instruct

Key findings

A single fine-tuned Phi-4 model achieved 99.6% token-level accuracy on a 100-million-token test set, outperforming an existing open-source benchmark detector.
The model family ensemble achieved 80.6% accuracy, outperforming the model-specific ensemble's 71.3% accuracy while using fewer detectors.
Both ensemble approaches achieved high recall up to 0.988 but lower precision because aggregating multiple detectors increased false-positive classifications of human text.

Summary drafted with AI from the full paper and may contain errors. See the full paper for details.

The rapid progress of large language models has enabled the generation of text that closely resembles human writing, creating challenges for authenticity verification in education, publishing, and digital security. Detecting AI-generated text has therefore become a crucial technical and ethical issue. This paper presents a comprehensive study of AI-generated text detection based on large-scale corpora and novel training strategies. We introduce a 1-billion-token corpus of human-authored texts spanning multiple genres and a 1.9-billion-token corpus of AI-generated texts produced by prompting a variety of LLMs across diverse domains. Using these resources, we develop and evaluate numerous detection models and propose two novel training paradigms: Per LLM and Per LLM family fine-tuning. Across a 100-million-token benchmark covering 21 large language models, our best fine-tuned detector achieves up to $99.6\%$ token-level accuracy, substantially outperforming existing open-source baselines.