Baid: A Benchmark For Bias Assessment Of Ai Detectors

Priyam Basu, Yunfeng Zhang, Vipul Raheja

Date

12/2025

AI Overview

Why it matters

AI-generated text detectors are increasingly used in education, yet they risk penalizing individuals based on their language background if deployed without systematic evaluations of fairness across sociolinguistic groups.

Study at a glance

Question: Do AI-generated text detectors perform unfairly across different demographic, educational, dialectal, and stylistic subgroups?
Setting and data: Benchmark of 208,166 document pairs across 7 bias categories and 41 subgroups, using datasets covering student essays, blogs, dialects, and political articles.
Method: Benchmark evaluation comparing four open-source AI text detectors across demographic, stylistic, and linguistic subgroups.
AI tools: GPT-4.1, Claude Sonnet 3.7, Desklib, E5-small, Radar, ZipPy

Key findings

AI detectors exhibited consistent disparities in detection performance across subgroups, notably showing low recall rates that risk misclassifying human writing from underrepresented groups as AI-generated.
Detector performance dropped substantially on dialectal and informal writing, with Desklib precision falling to 0.44 on Singlish and 0.16 on GenZ English.
Aggregate accuracy metrics masked substantial subgroup disparities, as detectors performed very differently across demographic groups, grade levels, and writing registers.

Summary drafted with AI from the full paper and may contain errors. See the full paper for details.

AI-generated text detectors have recently gained adoption in educational and professional contexts. Prior research has uncovered isolated cases of bias, particularly against English Language Learners (ELLs) however, there is a lack of systematic evaluation of such systems across broader sociolinguistic factors. In this work, we propose BAID, a comprehensive evaluation framework for AI detectors across various types of biases. As a part of the framework, we introduce over 200k samples spanning 7 major categories: demographics, age, educational grade level, dialect, formality, political leaning, and topic. We also generated synthetic versions of each sample with carefully crafted prompts to preserve the original content while reflecting subgroup-specific writing styles. Using this, we evaluate four open-source state-of-the-art AI text detectors and find consistent disparities in detection performance, particularly low recall rates for texts from underrepresented groups. Our contributions provide a scalable, transparent approach for auditing AI detectors and emphasize the need for bias-aware evaluation before these tools are deployed for public use.