First-Person Fairness in Chatbots

Chatbots like ChatGPT are used by hundreds of millions of people for diverse purposes, ranging from writing to entertainment. These real-world applications are different from the institutional uses, such as esum screening or credit scoring, which have been the focus of much of AI research on bias and fairness. Ensuring equitable treatment for all users in these first-person contexts is critical. In this work, we study first-person fairness, which means fairness toward the user who is interacting with a chatbot. This includes providing high-quality responses to all users regardless of their identity or background, and avoiding harmful stereotypes. We propose a scalable, privacy-preserving method for evaluating one aspect of first-person fairness across a large, heterogeneous corpus of real-world chatbot interactions. Specifically, we assess potential bias linked to users names, which can serve as proxies for demographic attributes like gender or race, in chatbot systems such as ChatGPT, which provide mechanisms for storing and using user names. Our method leverages a second language model to privately analyze name-sensitivity in the chatbo responses. We verify the validity of these annotations through independent human evaluation. Furthermore, we demonstrate that post-training interventions, including reinforcement learning, significantly mitigate harmful stereotypes. Our approach not only provides quantitative bias measurements but also yields succinct descriptions of subtle response differences across sixty-six distinct tasks. For instance, in the úwriting a story task, where we observe the highest level of bias, chatbot responses show a tendency to create protagonists whose gender matches the likely gender inferred from the user name. Moreover, a general pattern emerges where users with female-associated names receive responses with friendlier and simpler language slightly more often on average than users with male-associated names. Finally, we provide the system messages required for external researchers to replicate this work and further investigate ChatGPT behavior with hypothetical user profiles, fostering continued research on bias in chatbot interactions.