内容来源：https://research.google/blog/convapparel-measuring-and-bridging-the-realism-gap-in-user-simulators/

内容总结：

谷歌发布ConvApparel数据集与评估框架，量化AI用户模拟器的“真实性鸿沟”

2026年4月9日，谷歌研究院科学家Ofer Meshi与Sally Goldman团队发布了一项名为ConvApparel的新研究。该研究旨在解决当前基于大语言模型（LLM）的用户模拟器普遍存在的“真实性不足”问题，为训练更鲁棒的对话式AI系统提供了关键工具。

当前，对话式AI代理在处理复杂多轮任务方面已取得长足进步，但其在长对话中仍易出现遗忘约束或答非所问的情况。为高效训练和改进这些系统，业界常采用由LLM驱动的用户模拟器来替代成本高昂、难以规模化的人类实时测试。然而，现有模拟器常表现出与真实人类不符的行为，例如过度耐心、知识过于“百科全书化”或缺乏一致的个人偏好，这构成了显著的“真实性鸿沟”。

为量化并弥合这一鸿沟，研究团队构建了ConvApparel数据集及一套三支柱综合评估框架。该数据集专注于对话式推荐系统（CRS）场景，包含超过4000段、总计近1.5万轮的人类与AI在服装购物领域的多轮对话。

ConvApparel的核心创新在于其独特的“双代理”数据收集协议：参与者的购物请求被随机分配给一个“好”的AI助手（乐于助人、高效）或一个“坏”的AI助手（故意不配合、混淆关键词、检索能力降级）。这种设计捕获了用户从满意到极度沮丧的全方位体验。此外，数据集包含了每轮对话中用户自我报告的情感状态（如满意度、挫败感），为评估提供了宝贵的第一手真实基准。

基于此数据集，团队提出了评估模拟器真实性的三支柱框架：

1. 群体级统计对齐：检查模拟对话在对话长度、每轮词数、对话行为类型等统计指标上是否与人类对话分布一致。
2. 拟人度评分：训练一个自动判别器，以概率形式量化一段对话“像人”的程度。
3. 反事实验证：让仅在“好”代理数据上训练的模拟器去与从未见过的“坏”代理互动，检验其是否能像真实人类一样，表现出挫败感上升、满意度下降的合理适应性。

研究团队使用Gemini模型家族构建了三种代表性模拟器（基于提示、基于上下文学习、基于监督微调）进行测试。实验发现：

• 真实性鸿沟极易被检测，即使最佳模型也会因语法过于完美、话轮转换过于规律而暴露出非人痕迹。
• 数据驱动的方法在统计对齐上表现更优，但仍存在可检测的真实性差距。
• 在关键的反事实验证中，基于提示的基线模拟器面对“坏”代理时未能合理适应，而数据驱动的模拟器则能展现出类似人类的挫败行为变化，显示出更强的泛化能力。

这项研究指出，盲目依赖不真实的用户模拟器来优化AI代理，可能会损害其真实场景下的性能。ConvApparel数据集与评估框架为社区提供了系统化测量和缩小真实性鸿沟的工具。未来工作将聚焦于利用高保真模拟器从头训练CRS代理，并最终通过真实世界性能来闭环验证模拟器的有效性，以推动构建下一代真正可靠、实用的对话式人工智能。

中文翻译：

ConvApparel：量化并弥合用户模拟器的真实感差距
2026年4月9日
Ofer Meshi 与 Sally Goldman，Google Research 研究科学家

我们推出 ConvApparel——一个全新的人机对话数据集及综合评估框架，旨在量化基于大语言模型（LLM）的用户模拟器存在的“真实感差距”，并提升对话智能体的鲁棒性训练。

快速导读
现代对话式AI智能体通常能处理复杂的多轮任务，例如提出澄清性问题并主动协助用户。然而，它们在长程交互中常表现不佳，容易遗忘约束条件或生成无关回复。改进这些系统需要持续的训练与反馈，但依赖真人测试这一“黄金标准”成本极高、耗时漫长且难以规模化扩展。

作为一种可扩展的替代方案，AI研究界日益转向用户模拟器——即基于LLM、被明确指示扮演人类用户的智能体。然而，当前的LLM模拟器仍存在显著的真实感差距，表现为异常的耐心水平或不切实际（有时甚至如百科全书般）的领域知识。这好比飞行员使用飞行模拟器：最佳的模拟器应尽可能贴近现实，包含不可预测的天气、突发阵风甚至偶尔的飞鸟撞发动机事件。要弥合基于LLM的用户模拟器的真实感差距，我们首先需要量化这一差距。

在近期发表的论文中，我们正式介绍了ConvApparel——一个专为此目标设计的新型人机对话数据集。ConvApparel揭示了当前用户模拟的潜在缺陷，并为构建可信赖的AI测试者提供了路径。为全面捕捉人类行为（从满意到极度不满），我们采用了独特的双智能体数据收集协议：参与者被随机分配至一个有益的“友好”智能体或一个故意低效的“低质”智能体。结合包含群体统计、拟人度评分及反事实验证的三支柱验证策略，这一设计使我们能超越表面的简单模仿。

核心挑战
基于LLM的用户模拟器常表现出系统性偏离真人交互的行为，例如过度冗长、缺乏一致人设、无法表达连贯偏好、持有不切实际的“知识”以及异常耐心。由于大多数LLM被训练为高效助手，当要求其扮演不完美、易受挫的人类用户时表现不佳并不意外。若仅让对话智能体与这些不真实的模拟器交互训练，其在实际部署时很可能失败。

使用真实用户行为训练模拟器固然有效，但一个真正逼真的模拟器不应仅复现训练数据中的行为，还需对新奇、未见的情境（例如新对话策略）做出合理反应。这一点至关重要，因为模拟器的主要目标之一是帮助改进智能体，而改进过程常涉及测试与训练数据来源迥异的新智能体。过度拟合训练数据的模拟器对测试未经验证的新AI智能体毫无用处。这引出了一个关键的方法论挑战：如何检验模拟器的适应能力？

为此，我们引入了反事实验证的概念：若模拟用户遇到一个与训练中所见“友好”系统截然不同的低效系统，它会作何反应？通过评估模拟器应对意外糟糕或令人沮丧的对话智能体时的表现，我们可以判断其是否真正学会了合理的人类行为，抑或仅是盲目重复训练模式。

ConvApparel数据集与评估框架
对话式AI智能体最具前景的应用之一是对话式推荐系统（CRS），其中AI智能体作为复杂的决策支持系统，能够进行复杂推理与个性化引导。为建立CRS中人类行为的基线并支持新型反事实验证，我们构建了ConvApparel——一个包含服装购物领域超4000段人机多轮对话（总计近15000轮）的数据集。

ConvApparel的独特优势在于其双智能体数据收集协议。在参与者不知情的情况下，其购物请求被随机分配至两个不同的AI推荐器之一：

• “友好”智能体：被设定为利用强大搜索能力的高效购物助手。
• “低质”智能体：被明确设计为低效、偏离主题且令人困惑。它会微妙曲解关键词并使用故意降级的搜索检索功能。

这一双智能体设置是ConvApparel的关键设计特征，提供了两种受控环境，覆盖了从愉悦到极度不满的广泛用户体验。此外，ConvApparel包含细粒度的逐轮标注：我们要求参与者在对话每一轮后回溯报告其内部状态（如满意度、挫败感、购买意愿），从而提供了罕见的第一人称用户体验真实数据，用于验证实验设置与模拟行为。

基于这一丰富数据集，我们开发了包含三支柱的综合数据驱动框架以评估模拟器保真度，并比较了三种模拟器：提示式、上下文学习式及监督微调式（详见下文）。

1. 群体统计对齐：检验模拟对话在多项聚合统计指标（如对话长度、每轮词数、对话行为类型）上是否与人类对话匹配。
2. 拟人度评分：为捕捉细微风格差异，我们使用人类与模拟对话混合数据训练自动判别器，输出代表对话“拟人程度”的概率分数。
3. 反事实验证：利用双智能体数据，我们仅用“友好”智能体对话训练模拟器，再让其与未见的“低质”智能体交互。高保真模拟器应自然适应，表现出与人类相似的挫败感上升与满意度下降。

实验设计
我们将三支柱评估框架应用于基于Gemini模型系列构建的三种代表性LLM用户模拟器：（1）基于提示的模拟器（仅依赖高层行为指令，无特定训练）；（2）上下文学习模拟器（使用检索增强生成技术，每轮为模型提供ConvApparel中语义相似的人类对话示例）；（3）监督微调模拟器（通过在ConvApparel人机对话文本上直接训练Gemini 2.5 Flash模型，使其行为与目标人群深度对齐）。

每种模拟器需生成600段对话（与“友好”“低质”智能体各300段），以便与人类基线对比。

为保障研究伦理，我们确保对所有参与者完全透明并给予公平报酬。标注员为签署知情同意书的受聘人员，其薪酬高于所在国家生活工资标准。此外，标注员被明确要求以真实购物意图使用推荐系统，我们告知所有参与者其正在与开发中的实验原型交互，并明确提示系统可能表现不佳。

实验结果
实验得出多项重要发现：

1. 真实感差距极易检测：基于拟人度评分，训练后的判别器能准确识别近所有模拟对话为合成内容。即使最优的监督微调模型仍会产生细微痕迹（如完美语法和过度可预测的轮转模式）而暴露身份。
2. 数据驱动方法在统计对齐中胜出：在群体统计测试中，数据驱动模拟器（上下文学习式与监督微调式）持续优于简单提示基线，在冗长度和推荐接受率等行为分布上更贴近人类；但严格统计检验表明，即使这些更优模拟器仍存在持续的真实感差距。
3. 反事实验证展现鲁棒性：当与令人沮丧的“低质智能体”交互时，提示基线基本无法适应，仍保持非自然的礼貌与耐心。而数据驱动的上下文学习与监督微调模拟器表现出卓越的分布外泛化能力：尽管训练中从未接触“低质智能体”，它们仍真实调整行为，展现出显著更高的模拟挫败感与拒绝率。

结论
构建可靠的用户模拟器是开发下一代鲁棒、有益且高效的对话式AI的基础步骤。我们的研究强调，尽管基于LLM的用户模拟器前景广阔，盲目依赖它们仍存在重大风险。“真实感差距”持续存在，若为迎合不真实的模拟器优化AI智能体，可能损害其真实场景性能。

通过推出ConvApparel数据集及三支柱验证框架，我们为学界提供了严谨量化并最终弥合这一差距的必要工具。反事实验证证明，我们必须超越表面模仿，确保模拟器能逼真适应新型对话动态。我们邀请研究者和开发者探索ConvApparel数据集，运用本框架构建对话式AI未来所需的可靠合成用户。

未来方向
虽然实验表明数据驱动模拟器远优于提示式方法，但创建高真实感人工用户仍是开放挑战。我们的框架成功测量了真实感差距，但确定训练鲁棒对话智能体所需的具体保真度阈值仍是待解问题。

未来工作应聚焦于使用高保真模拟器从头训练并优化对话式推荐系统智能体，进而评估其真实场景性能。实现这一闭环将最终帮助我们量化构建高效、用户就绪型AI系统所需的“拟人度”标准。

致谢
本研究与合著者共同完成：Krisztian Balog, Avi Caciularu, Guy Tennenholtz, Jihwan Jeong, Amir Globerson 与 Craig Boutilier。

英文来源：

ConvApparel: Measuring and bridging the realism gap in user simulators
April 9, 2026
Ofer Meshi and Sally Goldman, Research Scientists, Google Research
We introduce ConvApparel, a new human-AI conversation dataset and a comprehensive evaluation framework designed to quantify the "realism gap" in LLM-based user simulators and improve the training of robust conversational agents.
Quick links
Modern conversational AI agents can typically handle complex, multi-turn tasks like asking clarifying questions and proactively assisting users. However, they frequently struggle with long interactions, often forgetting constraints or generating irrelevant responses. Improving these systems requires continuous training and feedback, but relying on the "gold standard" of live human testing is prohibitively expensive, time-consuming, and notoriously difficult to scale.
As a scalable alternative, the AI research community has increasingly turned to user simulators — LLM-powered agents explicitly instructed to roleplay as human users. However, modern LLM-based simulators can still suffer from a significant realism gap, exhibiting atypical levels of patience or unrealistic, sometimes encyclopedic knowledge of a domain. Think of it like a pilot using a flight simulator: the best simulators are as realistic as possible, with unpredictable weather, sudden gusts of wind, and even the occasional bird flying into the engine. To close the realism gap for LLM-based user simulators, we need to quantify it.
In our recent paper, we introduce ConvApparel, a new dataset of human-AI conversations designed to do exactly that. ConvApparel exposes the hidden flaws in today’s user simulation and provides a path towards building AI-based testers we can trust. To capture the full spectrum of human behavior — from satisfaction to profound annoyance — we employed a unique dual-agent data collection protocol where participants were randomly routed to either a helpful "Good" agent or an intentionally unhelpful "Bad" agent. This setup, paired with a three-pillar validation strategy involving population-level statistics, human-likeness scoring, and counterfactual validation, allows us to move beyond simple surface-level mimicry.
The challenge
LLM-based user simulators often exhibit behaviors that systematically deviate from genuine human interaction, such as excessive verbosity, lack of a consistent persona, inability to express coherent preferences, unrealistic “knowledge,” and unreasonable patience. Because most LLMs are trained to excel as helpful assistants, it’s not surprising that they perform poorly when tasked with playing the role of imperfect, easily frustrated human users. If we train our conversational agents to engage only with these unrealistic simulators, they may fail when deployed to actual users in the real world.
Using actual user behavior to train a simulator can be effective. However, a truly realistic simulator shouldn’t only reflect behavior drawn from its training data, but also react plausibly to novel, unseen situations (e.g., new conversational agent policies). This is crucial because one primary goal of simulators is to help improve the agent, which often includes experimenting with new agents that behave quite differently from the one used to generate the simulator's training data. A simulator that overfits to its training data is useless for testing new, unproven AI agents. This leads to a critical methodological challenge: how do we test a simulator's ability to adapt?
To solve this, we introduce the concept of counterfactual validation, which asks, how would a simulated user react if it encountered a frustrating system that looked nothing like the helpful ones it learned from during its (the user simulator’s) training? By evaluating how simulators handle unexpectedly bad or frustrating conversational agents, we can determine if they have actually learned plausible human behavior or if they’re just blindly repeating training patterns.
The ConvApparel dataset and evaluation framework
One of the most promising applications of conversational AI agents is Conversational Recommender Systems (CRSs), where an AI agent serves as a sophisticated decision-support system capable of complex reasoning and personalized guidance. To establish a baseline for human behavior in a CRS and enable this new type of counterfactual validation, we built ConvApparel, a dataset comprising over 4,000 human-AI multi-turn conversations (totaling nearly 15,000 turns) in the apparel shopping domain.
What makes ConvApparel uniquely powerful is its dual-agent data collection protocol. Unbeknownst to the participants, their shopping requests were randomly routed to one of two distinct AI recommenders:

• The "Good" agent: Prompted to be a helpful, efficient shopping assistant utilizing robust search capabilities.
• The "Bad" agent: Explicitly designed to be unhelpful, slightly tangential, and confusing. It subtly misinterpreted keywords and utilized intentionally degraded search retrieval.
  This dual-agent setup is the key design feature of ConvApparel. It provides two distinct, controlled environments, capturing a wide spectrum of user experiences ranging from delight to profound annoyance. Furthermore, ConvApparel includes fine-grained, turn-by-turn annotations. We asked participants to retrospectively report their internal states — such as satisfaction, frustration, and purchase likelihood — at every turn of their conversations, providing a rare ground-truth dataset of the first-person user experience necessary to validate both our experimental setup and the simulated behaviors.
  Using this rich dataset, we developed a comprehensive, data-driven framework consisting of three pillars to assess simulator fidelity. We compare three different simulators: Prompted, ICL, and SFT (details below).
• Population-level statistical alignment: We check if the simulated conversations match the human conversations with respect to various aggregate statistics, such as conversation length, words per turn, or the types of dialog acts taken (e.g., rejecting a recommendation).
  2. Human-likeness score: To capture subtle stylistic distinctions, we trained an automated discriminator on a mix of human and simulated conversations to output a single probability score representing how "human" a conversation feels.
  3. Counterfactual validation: Leveraging our dual-agent data, we train a simulator exclusively on conversations with the "good" agent, and then have it interact with the unseen "bad" agent. A high-fidelity simulator should naturally adapt, exhibiting a spike in frustration and decline in satisfaction similar to that humans displayed.
     Experiments
     We applied our three-pillar evaluation framework to three representative LLM-powered user simulators built using the Gemini model family: (1) a prompt-based simulator, which relied on high-level behavioral instructions without any specific training; (2) an in-context learning (ICL) simulator, which used retrieval-augmented generation to provide the model with semantically similar human conversation examples from the ConvApparel conversations at each turn; and (3) a supervised fine-tuning (SFT) simulator created by training a Gemini 2.5 Flash model directly on the ConvApparel human-AI transcripts to deeply align its behavior with the target population.
     Each simulator was tasked with generating 600 conversations, 300 with the "good" agent and 300 with the "bad" agent, allowing us to compare their performance against the human baseline.
     To ensure the ethical integrity of our study, we maintained full transparency and fair compensation for all participants. Raters were paid contractors who signed a consent form and received their standard contracted wage, which is above the living wage in their country of employment. Furthermore, raters were explicitly tasked with using the recommender as if they were intending to purchase, we informed all participants that they were interacting with an experimental prototype currently in development, explicitly noting that the system might exhibit suboptimal behavior.
     Results
     Our experiments yielded several fascinating insights:
  4. The realism gap is highly detectable
     Based on our human-likeness score, the trained discriminator confidently identified nearly all simulated conversations as synthetic. Even our best SFT models still produce subtle artifacts — flawless grammar and overly predictable turn-taking — that give them away.
  5. Data-driven methods win on statistical alignment
     In our population-level tests, the data-driven simulators (ICL and SFT) consistently outperformed the simple prompted baseline, closely mirroring human behavioral distributions in verbosity and recommendation acceptance rates; however, rigorous statistical tests reveal a persistent realism gap even for these better simulators.
  6. Counterfactual validation shows robustness
     When asked to interact with the frustrating “bad agent,” the prompted baseline largely failed to adapt, remaining unnaturally polite and patient. However, the data-driven ICL and SFT simulators demonstrated remarkable out-of-distribution generalization. Despite having never seen the "bad agent" in their training data, they realistically shifted their behavior, displaying noticeably higher levels of simulated frustration and rejection.
     Conclusion
     Creating reliable user simulators is a foundational step toward developing the next generation of robust, helpful, and effective conversational AI. Our research highlights that while the promise of LLM-based user simulators is massive, relying on them blindly carries significant risks. The "realism gap" is persistent, and optimizing AI agents to please unrealistic simulators could harm real-world performance.
     By introducing the ConvApparel dataset and our three-pillar validation framework, we provide the community with the tools necessary to rigorously measure and, ultimately, bridge this gap. Counterfactual validation proves that we must look beyond surface-level mimicry to ensure our simulators can realistically adapt to novel conversational dynamics. We invite researchers and developers to explore the ConvApparel dataset and utilize our framework to build the reliable synthetic users needed for the future of conversational AI.
     What's next?
     While our experiments show that data-driven simulators are vastly superior to prompt-based ones, creating a highly realistic artificial user remains an open challenge. Our framework successfully measures the realism gap, but determining the precise degree of fidelity needed to effectively train a robust conversational agent remains an open question.
     Future work should focus on using these high-fidelity simulators to train and refine CRS agents from scratch, and measuring the resulting real-world performance. Closing this loop will finally allow us to quantify the degree of “human-likeness” needed to build effective, user-ready AI systems.
     Acknowledgements
     This research was conducted in collaboration with our co-authors: Krisztian Balog, Avi Caciularu, Guy Tennenholtz, Jihwan Jeong, Amir Globerson, and Craig Boutilier.