Search and Filter

Consistently Simulating Human Personas With Multi-Turn Reinforcement Learning

Date

AI Overview

Why it matters

Simulating human users with language models enables scalable training and evaluation of interactive AI agents, but off-the-shelf models often contradict themselves, drift from assigned personas, and abandon role-appropriate behavior.

Study at a glance
  • Question: Can multi-turn reinforcement learning improve how consistently language models maintain simulated human personas across extended dialogues?
  • Setting and data: Simulated multi-turn dialogues across education, mental health, and open-ended conversation; human evaluation with 30 vetted adult participants via Qualtrics.
  • Method: Multi-turn reinforcement learning fine-tuning (Proximal Policy Optimization) evaluated against baselines using automated LLM-as-a-judge metrics and human validation.
  • AI tools: Llama-3.1-8B-Instruct, Gemma-2B-IT, Mistral-7B-Instruct, Meta-Llama-3.1-70B-Instruct, gpt-4o-mini, Qwen3-32B
Key findings
  • Fine-tuning with multi-turn Proximal Policy Optimization reduced persona inconsistency by over 55% across simulated tasks compared to the baseline model.
  • Prompt consistency improved with PPO fine-tuning by +20.6% in education, +37.6% in mental health, and +58.5% in open-ended conversation tasks.
  • Automated LLM judges achieved an average of 80.03% agreement with human annotators across consistency metrics, exceeding human-to-human agreement rates.

Summary drafted with AI from the full paper and may contain errors. See the full paper for details.

Large Language Models (LLMs) are increasingly used to simulate human users in interactive settings such as therapy, education, and social role-play. While these simulations enable scalable training and evaluation of AI agents, off-the-shelf LLMs often drift from their assigned personas, contradict earlier statements, or abandon role-appropriate behavior. We introduce a unified framework for evaluating and improving persona consistency in LLM-generated dialogue. We define three automatic metrics: prompt-to-line consistency, line-to-line consistency, and Q&A consistency, that capture different types of persona drift and validate each against human annotations. Using these metrics as reward signals, we apply multi-turn reinforcement learning to fine-tune LLMs for three user roles: a patient, a student, and a social chat partner. Our method reduces inconsistency by over 55%, resulting in more coherent and faithful simulated users.