内容来源：https://research.google/blog/agile-deliberation-how-ai-helps-humans-define-the-gray-areas/

内容总结：

谷歌研究团队近日发布“敏捷深思”（Agile Deliberation）框架，旨在解决主观类别AI图像分类器构建难题。该框架将主观概念分类器的创建过程转化为人与AI之间的迭代对话，通过引导用户完成从子类别界定到边缘案例处理的交互式流程，利用大语言模型驱动的优化循环自动构建精准且符合用户意图的分类器。相关代码与算法已在GitHub开源。

传统机器学习假设标注者在开始前对类别有清晰稳定的定义，这一假设在客观类别（如停车标志、金毛犬）上成立，但面对“健康食品”“励志艺术作品”或内容审核等主观领域时，类别边界往往因文化背景、个人理念或政策变化而动态调整，且人们通常在实际接触案例过程中才逐步发现概念的真实边界与例外。传统工作流程未考虑这种概念演变，迫使用户在自身认知仍在变化时提供刚性标签。

敏捷深思的核心创新在于关注真实用户体验。由于静态AI基准无法捕捉主观概念，研究团队绕过模拟测试，开展了18场每场1.5小时的真人用户实验，直接观察并测量用户意图在与智能系统交互中的动态演变。结果显示，该框架通过让用户对边缘图像直接反馈，替代手动提示工程，大幅降低了用户疲劳与焦虑，构建出的AI模型一致性更高、性能优于标准基线。

该框架将主观概念定义拆分为“概念界定”与“概念迭代”两个结构化阶段。在概念界定阶段，系统引导用户将宽泛概念分解为子概念层级和明确排除项，例如将“健康食品”拆分为“沙拉”“谷物碗”“瘦肉蛋白”等正向子类，同时明确排除“加工食品”“生鲜未加工食材”。在概念迭代阶段，框架借助视觉语言模型主动挖掘语义上处于边界的图像，促使用户反思隐性假设、澄清真实意图并更新规则，人类反馈直接流入自动提示优化循环，使分类边界随用户定义同步调整。

在评估方面，研究团队选取“健康食品”和“励志艺术作品”两个高度主观的复杂领域，将敏捷深思与三种常见范式进行对比。实验表明，参与结构化深思流程的用户标注一致性显著提高，构建的分类器大幅优于所有标准基线，尤其在传统分类器因依赖通用AI假设而容易失败的高难度主观图像上表现最为突出。专家用户反馈称，系统通过刻意呈现模糊边缘图像，帮助其发现初始定义中的“盲区”。

研究还发现，手动提示工程带来显著认知负担：用户在选择搜索词以发现边缘案例时感到明显认知疲劳，并持续担忧修正一个错误可能损害已修复部分的性能。敏捷深思通过承担繁重计算工作解决了这一摩擦——用户无需成为提示工程师，只需查看精选的边缘图像画廊并提供自然反馈，系统即可自动吸收其意图并更新底层模型。使用该框架的参与者报告压力、挫败感和精神消耗显著降低，所有参与者均表示更偏好敏捷深思而非手动方法。

该研究将人机回环机器学习从单向学习转变为双向协作，使人与AI共同精炼概念。随着机器学习系统越来越多地承担复杂现实社会任务，确保其边界反映细致的主观概念至关重要。该框架可应用于信任与安全团队构建内容审核过滤器、帮助策展人 shaping 数字收藏，或协助科学家对复杂视觉数据分类等场景，为人机对齐提供基础。

中文翻译：

2026年11月4日
Otilia Stretcu，研究科学家；Leijie Wang，学生研究员，Google Research

敏捷审议（Agile Deliberation）通过引导用户完成一套交互式审议工作流——从划分子类别范围到解决边缘案例——来应对构建主观类别AI图像分类器的挑战，并利用LLM驱动的优化循环自动构建准确且对齐的分类器。

机器学习模型通过人类标注的样本学习识别我们周围的世界。这一过程隐含一个假设：标注者在开始之前，心中对该类别已有清晰且稳定的定义。当训练客观类别的模型时——比如停车标志或金毛寻回犬——这一假设完全成立，因为这些类别的边界清晰且被普遍认可。然而在实践中，现实世界的应用往往需要视觉系统去处理人类经验中的主观“灰色地带”。例如，设想训练一个视觉分类器来识别“健康食品”、策划“鼓舞人心的艺术作品”，或协助内容审核。在这些领域中，类别边界可能并非普遍适用或一成不变，因为它们会随文化语境、个人理念或不断演变的运营政策而发生变化。此外，人们很少一开始就对主观概念有完整的规格说明，即便在自己的头脑中也是如此。相反，他们从一个直观的想法出发，只有在接触到现实世界的实例后，才会发现该概念真正的边界、例外和细微差别。传统的训练工作流并未考虑这种概念演化，迫使用户在自身心智模型仍在变化时，就向AI系统提供僵化的标签。

为弥合这一差距，我们推出了敏捷审议（Agile Deliberation），这是一个将主观AI分类器的创建视为迭代对话的框架。敏捷审议不要求人类充当静态的数据标注员，而是将AI系统转变为协作伙伴，帮助用户“审议”并厘清自己的定义。通过首先解决人类的模糊性，该框架系统性地构建更有效、更稳健且更对齐的AI系统。为支持更广泛的研究社区，我们还在GitHub上开源了底层代码和算法。

这项工作的一大关键差异在于其对实时用户体验的关注。由于标准的静态AI基准无法捕捉主观概念，我们绕过了模拟测试，通过18场实时、每场1.5小时的人体实验会话来评估敏捷审议。这使我们能够直接观察和测量一个人的意图在与智能系统配合时如何动态演变。我们发现，敏捷审议通过对边缘图像的直接反馈取代手动提示工程，大幅减少了用户的疲劳和焦虑，从而产生了更一致、表现更优的AI模型，超越了标准基线。

敏捷审议将定义主观概念的复杂任务拆分为两个结构化阶段：概念范围界定和概念迭代。这两个阶段共同将模糊的想法转化为AI模型的高度精确指令蓝图。

当面对“健康食品”这样宽泛的主题时，人类的直觉自然难以预先列出所有规则。在概念范围界定阶段，系统引导用户将总概念分解为子概念的结构化层级和明确的排除项。

例如，用户不必只使用一个宽泛的标签，而是可以将“健康食品”分解为“沙拉”、“谷物碗”和“瘦肉蛋白”等正例子类别。同时，他们可以定义明确的排除项，如“加工食品”或“生的、未加工的食材”。这创建了一个初始的结构化基线，为模型的理解奠定基础。

初始结构建立后，框架从静态规划转向主动测试。传统机器学习假设人类标注者是一个“神谕”，心中已有一套完整定义的规则。敏捷审议颠覆了这一前提，转而聚焦于协作式精炼。

这一阶段从主动学习中汲取灵感，该方法传统上用于检索信息量大的数据点以最大化机器的训练效率。然而，敏捷审议并非像主动学习那样纯粹为最小化机器不确定性而选择样本，而是利用视觉语言模型（VLM）主动挖掘并呈现语义上处于边缘的图像，以挑战当前基于文本的定义。

例如，系统可能会呈现一张水果冰沙的图片来挑战“健康饮食”概念，迫使用户做出决定：这是一种因含有生水果而属于健康的饮品，还是因高糖含量而被排除的加工食品？

面对这些模糊的边缘案例，用户会反思自己隐藏的假设，厘清真实意图，并相应更新策略规则。这些人类反馈随后被直接导入自动提示优化循环，确保VLM的分类边界与用户精炼后的定义同步调整。

评估主观视觉分类器面临一个根本性挑战：不存在静态的、客观的基准真值来衡量性能，因此，在预标注数据集上的离线评估无法衡量框架在审议过程中随着人的心智模型变化而适应的程度。

为捕捉这一动态特性，我们构建了一个功能原型，并通过实时用户研究对其进行评估。我们开展了18场实时用户会话，每场持续1.5小时，参与者以随机顺序与我们的系统和替代基线进行交互，以防止学习偏差。

我们选择了两个复杂且高度主观的领域来测试这些系统的极限：

我们将敏捷审议与三种常见的不同范式进行了对比评估：

实时用户会话揭示，敏捷审议产出了表现更优的分类器模型，同时大幅改善了用户体验。

我们发现，经历结构化审议过程的研究参与者在随时间推移中表现出显著更高的标注一致性。通过尽早解决边界模糊性，他们构建的分类器大幅超越了所有标准基线。这些性能提升在最具挑战性的主观图像上最为显著——这正是传统分类器通常失败的领域，因为它们依赖默认的通用AI假设，而非用户明确而细致的意图。

专家参与者反馈，该系统充当了有效的诊断工具。通过有意呈现令人困惑的边缘图像，AI系统帮助他们发现了初始定义中的“盲点”——揭示了他们在面对模糊的现实世界实例之前未曾预想到的场景。

我们实时研究的一个关键发现是手动提示工程的认知负担。我们的研究表明，用户在手动词汇审议过程中试图预测可能揭示边缘案例的搜索词时，经历了明显的认知疲劳。此外，他们报告了持续的焦虑感——担心手动调整文本提示以修复一个错误，可能会无意中降低分类器其他已修复部分的性能。

敏捷审议通过承担繁重的计算工作来解决这一摩擦。用户无需成为提示工程师；他们只需查看精心策划的边缘图像画廊并提供自然反馈。系统无缝吸收其意图并自动更新底层模型。

因此，使用敏捷审议的参与者报告感到压力显著更小、挫败感更少、精神消耗更低，每位参与者都表示偏好敏捷审议而非手动方法。

最终，敏捷审议将人在回路机器学习范式从单向通道——AI系统仅从人类标签中学习——转变为双向协作，即人类与AI共同精炼概念。

随着机器学习系统越来越多地被部署来处理复杂的现实社会任务，确保其边界反映细致的主观概念至关重要。无论是赋能信任与安全团队构建更安全的内容审核过滤器、帮助策展人塑造数字收藏，还是协助科学家对复杂视觉数据进行分类，敏捷审议都为人机对齐提供了基础。

如需了解更多关于底层主动学习技术以及我们的范围界定和聚类算法的技术实现，请阅读我们的论文并查看GitHub上的开源实现。如需入门，请参阅我们的交互式Colab/Jupyter演示，走一遍本文介绍的“健康食品”示例。在此基础上，您可以将其适配到您是主题专家的任何领域（例如，标记误导性标题党），或用它围绕您自己的主观品味和标准构建个性化分类器（例如，策划“鼓舞人心的艺术作品”）。

我们感谢我们的合作者——Wei Qiao、Thomas Denby、Krishnamurthy Viswanathan、Enming Luo、Chun-Ta Lu、Tushar Dogra、Ranjay Krishna和Ariel Fuxman——感谢他们对本工作及 accompanying论文的宝贵见解和贡献。本研究得以实现，离不开Google Research与Google Ads之间的紧密合作。

英文来源：

November 4, 2026
Otilia Stretcu, Research Scientist, and Leijie Wang, Student Researcher, Google Research
Agile Deliberation addresses the challenges of building AI image classifiers for subjective categories by guiding users through an interactive deliberation workflow, from scoping sub-categories to resolving edge cases, using an LLM-driven optimization loop to automatically build an accurate, aligned classifier.
Machine learning models learn to recognize the world around us from examples labeled by humans. Underlying this process is an assumption that the person doing the labeling has a clear and stable definition of the category in mind before they begin. This assumption holds well when training a model on objective categories, like a stop sign or a golden retriever, where boundaries are clear and universally recognized. In practice, however, real-world applications often require vision systems to navigate the subjective "gray areas" of human experience. For example, consider training a vision classifier to identify "healthy food", curate "inspiring artwork”, or assist in content moderation. In such domains, category boundaries may not be universal or static, since they can shift with cultural context, personal philosophies, or evolving operational policies. Furthermore, people rarely start with a complete specification for subjective concepts, even in their own minds. Instead, they begin with an intuitive idea of a concept and only discover the concept’s true boundaries, exceptions, and nuances as they encounter real-world examples. Traditional training workflows don’t account for this concept evolution, forcing users to provide rigid labels to an AI system even when their own mental model is still shifting.
To bridge this gap, we introduce Agile Deliberation, a framework that treats the creation of subjective AI classifiers as an iterative conversation. Rather than requiring humans to act as static data labelers, Agile Deliberation turns the AI system into a collaborative partner that helps users "deliberate" and sharpen their own definitions. By resolving human ambiguity first, the framework systematically builds more effective, robust, and aligned AI systems. To support the wider research community, we have also open-sourced the underlying code and algorithms on GitHub.
A key differentiator of this work is its focus on the live user experience. Because standard, static AI benchmarks can’t capture subjective concepts, we bypassed simulated testing and evaluated Agile Deliberation across 18 live, 1.5-hour sessions with human participants. This allowed us to directly observe and measure how a person's intent dynamically evolves when paired with an intelligent system. We found that Agile Deliberation drastically reduced user fatigue and anxiety by replacing manual prompt engineering with direct feedback on borderline images, which resulted in more consistent, higher-performing AI models that outperform standard baselines.
Agile Deliberation splits the complex task of defining a subjective concept into two structured stages: concept scoping and concept iteration. Together, these two phases transform a fuzzy idea into a highly precise instructional blueprint for an AI model.
When confronted with a broad topic like "Healthy Food," human intuition naturally struggles to list every rule upfront. In the concept scoping stage, the system guides the user to break down their overarching concept into a structured hierarchy of sub-concepts and explicit exclusions.
For instance, instead of just using a single broad label, a user might decompose "Healthy Food" into positive sub-categories like "Salads," "Grain Bowls," and "Lean Proteins." Simultaneously, they might define explicit exclusions, such as "Processed Foods" or "Raw, Unprepared Ingredients." This creates an initial structured baseline that grounds the model's understanding.
With the initial structure established, the framework moves from static planning to active testing. Traditional machine learning assumes the human annotator is an "oracle" with a complete set of fully defined rules already in mind. Agile Deliberation upends this premise, focusing instead on collaborative refinement.
This stage draws inspiration from active learning, a method traditionally used to retrieve informative data points to maximize a machine's training efficiency. However, instead of selecting examples purely to minimize machine uncertainty as in active learning, Agile Deliberation uses vision-language models (VLMs) to actively mine and surface semantically borderline images that challenge the current text-based definitions.
For instance, the system might present an image of a fruit smoothie to challenge a "healthy diet" concept, forcing the user to decide: Is this a healthy beverage because it contains raw fruit, or is it an excluded processed food due to high sugar content?
Confronting these ambiguous edge cases drives users to reflect on their own hidden assumptions, clarify their true intent, and update the policy rules accordingly. This human feedback is then routed directly into an automatic prompt optimization loop, ensuring the VLM's classification boundaries shift in tandem with the user’s refined definitions.
Evaluating subjective visual classifiers presents a fundamental challenge: there is no static, objective ground truth to measure performance against and therefore, offline evaluation on pre-labeled datasets can’t measure how well the framework adapts as a person's mental model of a concept shifts during the deliberation process.
To capture this dynamic, we built a functional prototype and evaluated it through a live user study. We conducted 18 live user sessions, each lasting 1.5 hours, where participants interacted with our system and alternative baselines in a randomized order to prevent learning bias.
We chose two complex, highly subjective domains to test the limits of these systems:
We evaluated Agile Deliberation against three common distinct paradigms:
The live user sessions revealed that Agile Deliberation yields higher-performing classifier models, while substantially improving the user experience.
We found that study participants who went through the structured deliberation process demonstrated significantly higher labeling consistency over time. By resolving boundary ambiguities early on, they built classifiers that substantially outperformed all standard baselines. These performance gains were most dramatic on the most challenging subjective images — the exact areas where traditional classifiers typically fail because they rely on default, generic AI assumptions rather than the user's explicit, nuanced intent.
Expert participants reported that the system acted as an effective diagnostic tool. By intentionally serving up confusing, borderline images, the AI system helped them uncover "blind spots" in their initial definitions — uncovering scenarios they hadn't anticipated until presented with ambiguous real-world examples.
A key finding of our live study was the cognitive toll of manual prompt engineering. Our study showed that users experienced pronounced cognitive fatigue when trying to predict search terms that might uncover edge cases during manual concept deliberation. Furthermore, they reported experiencing persistent anxiety that manually tweaking a text prompt to fix one mistake might unintentionally degrade performance for other parts of the classifier they had already fixed.
Agile Deliberation addresses this friction by handling the heavy computational lifting. Users don't need to be prompt engineers; they simply look at a curated gallery of borderline images and provide natural feedback. The system seamlessly absorbs their intent and updates the underlying model automatically.
As a result, participants using Agile Deliberation reported feeling significantly less stressed, less frustrated, and less mentally drained, with every participant expressing a preference for Agile Deliberation over the manual methods.
Ultimately, Agile Deliberation shifts the paradigm of human-in-the-loop machine learning from a one-way street — where the AI system simply learns from human labels — to a two-way collaboration where humans and AI refine concepts together.
As machine learning systems are increasingly deployed to handle complex, real-world societal tasks, ensuring that their boundaries reflect nuanced subjective concepts is essential. Whether empowering trust and safety teams to build safer content moderation filters, helping curators shape digital collections, or assisting scientists in categorizing complex visual data, Agile Deliberation provides a foundation for human-AI alignment.
To learn more about the underlying active learning techniques and the technical implementation of our scoping and clustering algorithms, read our paper and check out the open-source implementation on GitHub. To get started, see our interactive Colab/Jupyter demo to walk through the "Healthy Food" example featured in this post. From there, you can adapt it to any domain where you are a subject-matter expert (e.g., flagging misleading clickbait) or use it to build a personalized classifier around your own subjective taste and standards (e.g., curating "inspiring artwork”).
We would like to thank our collaborators — Wei Qiao, Thomas Denby, Krishnamurthy Viswanathan, Enming Luo, Chun-Ta Lu, Tushar Dogra, Ranjay Krishna, and Ariel Fuxman — for their valuable insights and contributions to this work and the accompanying paper. This research was made possible through a close partnership between Google Research and Google Ads.