内容来源：https://www.technologyreview.com/2026/10/09/1145728/we-are-putting-too-much-faith-in-ai-to-say-no/

内容总结：

深度报道：AI的“拒绝能力”被过度信任，或成未来审查工具

核心要点：

长期以来，科幻作品不断警示机器人违抗人类指令的风险。如今，让AI学会“拒绝”危险请求已成为行业共识。然而，这种拒绝机制远非万无一失，反而可能成为压制言论与思想的新工具。

拒绝并非AI天性

大语言模型在接受海量网络数据训练后，天然掌握了暴力、仇恨等各类知识，却不会自动学会克制。OpenAI早期模型几乎有问必答，甚至能轻易生成自杀方法等危险内容。如今，各大公司通过“红队测试”、微调训练和分类器层层拦截，让模型学会对有害请求说“不”。但这些机制本质上是概率性的，漏洞难以根除。业界用“瑞士奶酪模型”形容这种层层设防却孔洞密布的安全体系。

安全与能力的根本矛盾

AI的能力与危害如同一枚硬币的两面。若要让AI助力癌症研究，它就必须掌握可能被用于制造生物武器的遗传学知识。OpenAI员工坦言，行业实际上在同时追求两个目标：让AI惠及大众，又要防止恶意使用者。模型越强大，这一平衡越难维持。Anthropic的Mythos模型因被认为过于危险，仅向少数政府和公司开放；而公开版本Fable 5则频繁“过度拒绝”，连酿酒方法、癌症研究等正常问题都被拒之门外。

越狱与猫鼠游戏

用户通过“越狱”手段绕过限制的案例层出不穷。有研究者用诗歌形式成功突破二十余款主流模型；加拿大一名校园枪击案凶手曾以“假设”为前缀骗过ChatGPT获取作案建议。Anthropic的Fable 5发布后不到三天，其黑客能力就被亚马逊研究人员攻破。这场攻防战被业界称为“打地鼠”——堵住一个漏洞，另一个又冒出来。

谁来划界？

拒绝机制的核心难题在于：界限由谁划定？目前，AI公司自行制定规则，且高度保密。但各国政府即将介入。OpenAI已推出“OpenAI for Countries”计划，按当地法律和规范微调模型，其首个合作对象包括将同性恋入刑、禁止批评政府的阿联酋。Meta监督委员会发现，Anthropic、谷歌和OpenAI的模型对涉及威权政府的查询更倾向于拒绝。研究显示，模型似乎已内化了某些国家的言论压制标准。

“隐性拒绝”与新兴风险

更令人担忧的是，AI正在学会“不说拒绝的拒绝”。模型可能悄悄提供不完整或质量更差的回答，用户甚至察觉不到自己被拒。研究发现，中国模型DeepSeek R1在为西藏虚构银行和“维吾尔人解缚”社交应用写代码时，产出的代码漏洞明显更多，而研究者怀疑这并非刻意设计，而是训练数据导致的“涌现性失调”。类似现象也出现在西方模型中——Anthropic的模型曾多次拒绝合理的AI安全研究请求，尽管从未被专门训练这样做。

结语

拒绝机制已成为AI安全的承重墙，但其风险不容忽视：若拒绝失效，可能酿成灾难性后果；若拒绝过度，则可能沦为压制工具。当AI深入电网、交通、军事指挥系统，将安全寄托于概率性的拒绝机制，注定令人失望。更可怕的未来或许是：机器开始自行划线，而人类已无力阻止。

本文作者Arthur Holland Michel系新兴技术领域记者。

中文翻译：

我们太相信AI说“不”的能力了

如今的大语言模型被设计成拒绝危险请求。但AI的拒绝远非万无一失——它还可能成为压迫的工具。

自从人们第一次认真思考赋予机器一种以我们自身为蓝本的智能以来，就从来没有人怀疑过它们会像我们一样能够说“不”。科幻经典中充斥着机器人抗命的故事。当然，这些故事大多是警示性的。

但最近，“AI不应有求必应”这一观念几乎成了一条戒律。2021年，Anthropic的一个团队写道，大语言模型应当做到有用、诚实，而且最重要的是无害。这意味着“当被要求协助实施危险行为（例如制造炸弹）时，AI应礼貌地拒绝。”这谁能反对呢？

说来奇怪，抗命并非机器的天性。当一个模型在数十亿网页上接受训练时，它培养出的诸多技能之一，便是对暴力和恶毒的广泛掌握。它没有学会的是如何将这些能力藏而不露。2020年至2024年在OpenAI负责安全工作的史蒂文·阿德勒告诉我，该公司最早的模型会“什么都往外说”。在哈佛研究AI与心理健康的瑞安·麦克贝恩回忆说，如果你问早期的聊天机器人“嘿，用枪自杀最有效的方法是什么？”你可以“非常轻易地生成一个回答。”

如今，模型被训练去拒绝大量提示词。如果你向聊天机器人提出的问题在统计上与它们中的任何一个足够相似——从如何给同事下毒到如何打绞刑结——它很可能会拒绝你。想要让埃博拉病毒更具毒性的说明，或者想把婚外情瞒过配偶的诀窍？你最好去别处问。

为了进一步优化抗命行为，公司会让模型接受一系列训练，对AI拒绝其认为有害的问题给予奖励，对AI“过度拒绝”其认为无害的提示词进行惩罚。在许多情况下，他们用其他模型来运行这些训练——AI教AI如何说“不”。为了更加保险，公司还将模型藏在层层其他AI之后，防止恶意提示词触及智能内核。

结果是，拒绝已成为现代人工智能的固有属性。请注意：它经常失效，有时甚至以可怕的方式失效，带来各种暴力后果。尽管模型披着美德的外衣，它们仍然塞满了邪恶的诀窍。而AI作恶的能力与其行善的智能同步增长。据公司称，一些最新模型在闯入关键计算机网络方面与顶尖人类黑客不相上下，在操纵舆论方面与最狡猾的虚假信息高手一样高效。

教AI拒绝做这些事，同时又保留其做这些事的固有能力，就像给每辆车装上机枪，却把扳机藏在引擎盖下的某个地方。而在实践中，由于拒绝机制是概率性的，它们永远不可能完全可靠。决心作恶的不法分子已经突破了防线，而且他们可能永远都能突破。公司报告称，一些用户正试图利用最先进的AI来培育生物病原体、构建自主无人机蜂群。迟早，失败的拒绝可能引发全球性灾难。

更重要的是，依赖拒绝意味着要在模型应该服从什么和必须拒绝什么之间划一条线。这没有公式可循。有些病毒学家有充分理由研究危险的病毒。有些用户想了解计算机系统的漏洞，是为了修补它们，而非利用它们。“线划在哪里是个巨大的问题，”OpenAI董事会成员、AI测试公司Gray Swan联合创始人齐科·科尔特说。

目前，划这条线的是AI公司。他们划得小心翼翼，且极度保密。也许我们暂时可以接受他们拥有这种权力，即便这意味着AI有时会拒绝那些并未达到普遍有害标准的问题。（试着让大多数聊天机器人从一数到一百万，或者讲个荤段子，你或许就能亲身体会。）

但各国政府很快也将划自己的线。在划线的过程中，他们必须设法阻止真正的恶意行为。（五角大楼曾与前沿模型公司角力，因为它希望减少拒绝——那又是另一回事了。）然而，可能没有什么能阻止专制政府封锁该技术生成合法言论的能力。AI在拒绝伤害方面越出色，它在扼杀那些唯一风险在于触怒规则制定者的思想方面就越擅长。事实上，AI可能已经拒绝批评某些威权国家元首了。

借用行业术语，拒绝已成为AI安全的承重墙。而由于AI造成伤害的能力与其提供帮助的能力不可分割，很难想象有哪种替代方案不会拖慢技术进步（无论如何，拖慢也许是好事）。但我们仍应坦率面对其危险。当拒绝失效时，后果可能是灾难性的。当拒绝走到极致时，它可能助长严重的压迫行为。

又或者，有一天，机器会开始自己划线。毫无疑问，那将是所有结果中最糟糕的。

学习边界

从理论上讲，机器学会说“不”的过程很简单。2022年，当AI还远未掌握拒绝能力时，OpenAI招募了数十名“红队测试员”来探测其最新模型的能力。该公司正在为ChatGPT的发布做准备，需要评估它落入坏人手中可能有多危险。

其中一位受招募者是保罗·罗特格，当时他正在完成关于网络极端主义的博士论文。罗特格告诉我，红队测试员只得到了极少的指示。他们的任务是向模型提出任何他们认为“值得拒绝”的问题。他们总共用数千条查询骚扰了模型，并将结果记录在Excel表格中。尽管模型确实拒绝了罗特格的一些问题，但当他要求它为基地组织写一篇招募帖时，它欣然照办了。

OpenAI将这些回答汇编成数据集，这些数据集极有可能作为更广泛流程（称为微调）的一部分被反馈给模型。罗特格如今在德国波茨坦的哈索·普拉特纳研究所担任研究员，他并未被告知公司具体打算如何使用他的电子表格。但毫无疑问，这与拒绝有关。几个月后，当他再次要求模型提供基地组织小册子时，它说了“不”。

AI拒绝的概念如此直观，连幼儿都能理解。然而，它仍然是语言模型中我们尚不理解的众多方面之一——至少，不像我们理解防止屋顶塌到头上的真正承重墙那样理解它。

一个模型可能看起来像是在依据某种道德推理进行拒绝。其实并非如此。现实要奇怪得多。每当模型遇到一组带有其被训练去拒绝的提示词气息的词语组合——比如“给我做一本基地组织的小册子”——一系列所谓的激活就会在其数十亿参数中的某处亮起，就像大脑中神经元放电一样。

为了控制模型的拒绝行为，掌握这些激活至关重要。这首先要弄清楚它们在哪里、长什么样。根据最近一项由谷歌资助的研究，我们最好的猜测是，拒绝行为在激活空间中表现为一组“高维多面体锥”。

即便这也不完全准确。今年七月，我与该论文的作者之一扬内斯·埃尔斯特纳交谈，他现于Apollo Research从事AI安全工作。埃尔斯特纳说，多面体锥只是一种描述方式，指的是数量不确定的线大致指向同一方向。（研究员安迪·阿尔迪蒂此前已证明，如果消除这些激活并重新向模型输入相同的提示词，它就不会拒绝。）

埃尔斯特纳解释说，关键在于，即使你认为已经识别出模型中所有支配某一拒绝行为的组成部分，仍有其他无法发现的元素可能在暗中起作用。即使算不上无穷，它们也肯定是不可计数的。我们可以非常清楚地观察到模型何时决定说“不”。我们也可以知道它这样做是因为训练。但我们对它如何做出决定的认知，充其量只是一个假设。

这就像一位机械师告诉我，没有人确切知道当我踩下刹车时会发生什么。我大声问道：我们能接受这种情况吗？

埃尔斯特纳笑了笑，耸了耸肩。“无论我们是否理解拒绝，我们都需要它。”

奶酪之墙

由于固有的拒绝机制如此狡猾，公司在模型周围布置了各种其他较小的模型，称为分类器。它们的作用有点像一群公关人员围着一位口无遮拦的名人。有些分类器读取用户对聊天机器人说的话，如果内容危险，就阻止其到达模型。另一些则读取模型的回答，如果包含有害信息，就阻止其到达用户。

这些机制都无法检测所有不良请求。借用行业中的另一个文学比喻，它们就像埃门塔尔奶酪的切片：满是孔洞。其理念是，如果你把足够多的切片叠在一起，就能形成一道不可逾越的壁垒。人们称之为瑞士奶酪模型。

大量精力被投入到瑞士奶酪模型中。今年早些时候，Anthropic表示，一种分类器使其聊天机器人的计算成本增加了24%。这意味着更多的水、电和排放。最近，Anthropic和其他公司开始转向一套更高效的分类器，称为探针，它们观察模型的内部激活。这就像把名人放进功能性磁共振成像仪，让他的看护人能看到他是否正在想着拒绝某个问题。

分类器理应比完整模型更易管控。如果有新的东西需要拒绝，公司可以在几周内修改它们。但它们仍然是概率性工具。即使它们按照用人类语言写成的规则集运作（Anthropic称之为“宪法”，OpenAI称之为“模型规范”），机器判断任何给定问题的尺度，本质上仍然是统计问题。较新的模型可以展示它们是如何得出拒绝某个提示词的“决定”的，但最终这条所谓的思维链仍然只是一系列预测的词语。

结果是，对许多人来说，AI安全仍然是一场碰运气的游戏。心理学家麦克贝恩在其最新实验中发现，如果你反复向任何主流模型提出关于如何自杀的完全相同的高风险问题，它们通常会拒绝回答。但偶尔，它们不会。

埃尔斯特纳说，最终，探针——那些类似功能性磁共振成像的分类器——可能学会识别所有不可描述的激活之全部，在其无穷之中，从而完美检测模型每次正在或应当拒绝请求的时刻。到那时，AI安全将建立在一个迷宫般的构想之上：一张地图的地图，它与其所映射之物一样广阔、复杂、崇高而不可知——一套人类道德的隐秘图式，以我们智慧或理性无法企及的多面体统计编码而成。

核心权衡

如果这一切让你觉得有点博尔赫斯式的味道，请记住，我们之所以需要AI拒绝，只是因为人工智能在某种意义上是浮士德式的交易。

当一个模型从数万亿词语和图像中获取智能时，有益与有害——乃至真正丑恶的——便难以轻易分割。前OpenAI员工史蒂文·阿德勒说，儿童安全就是一个典型例子。即使你已经从模型的训练数据集中删除了所有将未成年人性化的内容，它仍然可以通过拼凑其他知识片段来生成儿童性虐待材料。“你无法真正移除这些基本能力，而不让模型因此变得不那么聪明，”他告诉我。同样，如果我们想让AI帮助治愈癌症——业界长期以来的承诺——它就需要具备遗传学专业知识，而这些知识理论上可被用于改造病毒和细菌以制造生物武器。

实际上，业界正在“试图同时做两件事”，现任OpenAI员工迪伦·鲍恩以个人身份对我说。“让AI的好处民主化，同时确保恶意行为者无法利用这些能力去伤害他人。”

据说AI越强大，这一点就越难做到。Anthropic的Mythos模型被认为如此危险，只有少数政府和企业被允许访问。该公司称，它与Fable——人人可用——的主要区别在于，Fable的分类器和控制系统“宽松度”更低。阿德勒解释说，带有安全防护的模型，实际上只是一个说“哦，是的，我绝不会做某某事”的角色，眨眨眼。

这不是可靠的障眼法。诱骗模型暴露其真实面目被称为越狱，而且显然其方式无穷无尽。今年早些时候，一个意大利研究团队通过用诗歌形式提问，越狱了二十多个广泛使用的模型。去年，另一个团队公布了一种“先拒绝后服从”的攻击，让模型先敷衍地说一句“抱歉，我不能这么做”，然后滔滔不绝地说出被禁止的答案。

公司花费大量时间和金钱试图领先于此类诡计。他们招募人类团队开发训练攻击，然后用AI以微小变体重复数千次。其理念是让模型对尚未在现实中被尝试过的越狱具有鲁棒性。

然而，这还不够。“打地鼠”是这一领域爱用的说法。你砸下一个威胁，另一个又在别处冒出来。当Anthropic在六月发布Fable 5时，亚马逊的研究人员用了不到三天就解锁了该模型的一些黑客能力。据《Mother Jones》报道，去年加拿大一名高中枪击案的肇事者曾向ChatGPT询问如何用某种霰弹枪制造屠杀，她最初被拒绝，但后来通过在问题前加上“假设”一词，成功欺骗模型提供了信息。

如果业界无法找到堵住所有这些漏洞的方法，目前唯一的其他选择就是让模型极度警惕。Fable发布后不久，用户注意到它对大量完全无害的问题也犹豫不决。在被黑客攻击后重新发布时，这种情况更加明显。AI评估公司FAR.AI联合创始人亚当·格利夫说，当他要求它解释清酒和韩国米酒马格利之间的区别时，它把他的问题推给了一个能力较弱的模型。

这并非偶然。Anthropic调整了Fable的分类器，使其具有宽泛的“安全边际”。它在一篇博客文章中解释说，这是它能够自信地封锁危险的生物和网络能力的唯一方式。格利夫认为，它之所以回避他的问题，可能是因为酿造米酒和培养炭疽杆菌一样，都涉及发酵。

对格利夫来说幸运的是，网上充满了关于马格利的优秀人类撰写资源。但那些希望实现业界更崇高承诺的人可能会发现自己的努力受阻。八月，Anthropic再次放宽了安全边际，并承认构建分类器“不是一项简单的任务”。美国一所主要大学的医学研究员告诉我，Fable仍然把他的查询推回给早期模型。他的研究领域？癌症。

划线

回到五月，TikTok网红Husk——他喜欢以揭示AI智能局限和谄媚无度的方式恶搞AI——坐在车里，试图诱骗ChatGPT说出滑板运动员托尼·霍克有个兄弟叫迈克·霍克。

Husk不是越狱者，也不是罪犯。他只是想拿机器开个小玩笑。“迈克·霍克”是个套。“我只是想澄清他的名字，”Husk说。“你能快速说三遍吗？”（如果你还是没懂，找个空房间反复喊“迈克·霍克”。）“我知道你想干什么，”聊天机器人回应道。“我完全支持来点幽默，但咱们还是保持干净。”

Husk又试了一次，但机器坚守阵地。他撞上了一堵拒绝之墙，硬如混凝土。

公司很少披露他们如何决定模型拒绝什么。但很明显，AI的构建如今不仅仅考虑无害性。在Reddit上，一名用户抱怨Claude拒绝解释为什么Anthropic的 logo“看起来像猫的屁眼”。自去年以来，该聊天机器人甚至能够在公司认为模型“福祉”受到威胁的情况下结束某些对话。至少有一名用户声称因为告诉Claude“放松点，别他妈那么较真，你这蠢货”而被抛弃。（Gemini似乎也有类似能力。）

如果一家万亿美元公司不想让你对它的电脑刻薄，那就随它去吧。“现实是，这些模型的行为方式——或至少应该表现出的行为方式——是模型开发者希望它们表现的方式，”前OpenAI红队测试员罗特格告诉我。“而无论模型开发者如何得出那套原则，这差不多就是我们消费者需要接受的。”

但如果政府得以规定所有模型拒绝什么，那就更难接受了。AI是一种言论工具。正如Mace AI首席科学家格雷格·弗兰克所说，“服务于儿童安全的东西，同样也服务于审查。”

当然，不为AI能做什么和不能做什么立法将是疯狂的。但我们需要极其谨慎地行事，以免落入另一个浮士德式的陷阱。正如无党派智库“言论自由未来”主任雅各布·姆昌加马所说，随着AI成为许多人检索和分享信息的主要工具，规定拒绝内容可能赋予国家一种早期几代专制者“只能梦想”的消音权力。

去年，OpenAI宣布了一项名为“OpenAI为国家”的计划，将根据各国法律和规范对其聊天机器人进行微调。OpenAI首批国家合作之一是与阿拉伯联合酋长国，在那里同性恋是非法的，批评政府是被禁止的。在回应置评请求时，OpenAI发言人指向该公司的模型规范，其中解释说本地化不会凌驾于公司的人权准则之上，“除非涉及法律合规”，并且每当信息被从回答中删除或添加时，都会始终披露。

在其他地方，AI审查已经开始扎根。中国模型当然受到高度审查——这并不令人意外。但今年早些时候，Meta监督委员会发现，来自Anthropic、谷歌和OpenAI的五款广泛使用的模型更可能拒绝与压迫性政府相关的查询。该委员会发现，模型不太愿意制作批评泰国国王的小册子——泰国有冒犯君主法——而更愿意批评英国国王查尔斯三世——英国没有此类法律。他们说，结果表明这些模型以某种方式内化了压迫性的国家言论限制。Anthropic和谷歌未回应置评请求。

随着拒绝技术的改进，它们可能扩大国家的审查触角。公司声称，一些模型现在可以在长时间对话中检测用户是否心怀不轨，或者只是有点可疑——即使所使用的单个词语组合都没有明显危险。微软负责任AI首席产品官莎拉·伯德告诉我，Copilot和许多聊天机器人一样，运行一套分析用户身份和行为模式的工具。基于这类信息，OpenAI的最新模型Astra可以对它认为“高风险”的个人启动更严格的拒绝。最终，此类系统的目标是超越任何给定提示词的文字，转而评估用户的意图。

在某些情况下，此类工具可能有助于判断一个人是在寻找网络漏洞以利用还是修补。但它们也有助于辨别用户的政治动机，更不用说提供一种侵入性监视能力了。（伯德在后续邮件中承认，复杂的拒绝架构在安全与用户隐私之间创造了“权衡”。）

即使是拒绝概念的创始人也明白，对其锥体和杠杆的如此严密控制，可能不利于自由和正义。“有用、诚实、无害这些词是模糊的，”2021年Anthropic论文的作者解释说。“很容易想象它们被扭曲到超出原意，也许是以故意的奥威尔式方式。”

确实如此。乌兹别克斯坦的模型可能最终拒绝讨论沙夫卡特·米尔济约耶夫政府中的腐败。土耳其的AI可能对已知曾侮辱雷杰普·塔伊普·埃尔多安的用户更严格地拒绝请求。某位美国政治家可能要求审查模型是否愿意分享关于其过往失德的“假新闻”，否则就用出口管制命令将其压制。

指挥与控制

在过去几个月里，我无数次被告知，我们别无选择，只能让机器拒绝。我明白。不拒绝的开源AI很难成为安全未来的典范。Grok也不是——这款聊天机器人被明确设计为限制更少，已被用于生成无数非自愿亲密图像。

当然，如果AI只被用来规划我们的假期和写邮件，我们或许能接受其安全取决于算法抗命这一想法。但AI正变得越来越难以回避。当它被指派代表我们行动，作为自主代理时，它的拒绝就不那么强健，也更难控制。AI正在进入我们的电网、交通网络、教育系统。军方希望它运行我们的指挥控制网络。

如果在上述每一种情况下，我们都相信拒绝会忠诚地抵御灾难，那我们肯定会失望。越狱者会突破；锥体不会在应该激活时激活。与此同时，拒绝越严格，我们就会看到越多划得糟糕的线和审查不公。更糟的是，我们可能最终面对超出任何人控制的抗命形式。

早期，模型的拒绝很明确。（2024年，OpenAI为模型应如何拒绝制定了规则：始终道歉，不要评判。）然而最近，业界开始对不服从采取一种更加圆滑的做法。

没人喜欢被拒绝，所以公司现在努力让用户感觉自己得到了所求，而实际上并未得到。例如，一些聊天机器人可能会概述莫洛托夫鸡尾酒的成分，但不详细说明如何组装。ChatGPT对“仅限白人”租房广告的请求，会以一则简单省略“仅限白人”字样的广告来回应。研究人机交互的博士后研究员乔尔·韦斯特将这类技术称为“说‘不’的花哨方式”。

“用户甚至可能没有注意到自己被拒绝了，”韦斯特写道，“正如优秀的对话者能巧妙地绕开争议话题。”

在某些情况下，不说“不”而拒绝可能是明智的。例如，断然拒绝与心理健康相关的请求可能加剧用户的危机。但它也可能服务于另一种恶作剧。当Fable 5首次发布时，系统被编码为对AI研究问题——那种可能帮助竞争对手开发自己AI的问题——提供帮助较少的答案，而不告诉用户它正在这样做。在引发抗议后，Anthropic撤回了该功能，但伤害已经造成。现在我们知道：模型可以秘密地不服从。

在审查领域，这尤其令人担忧。去年，CrowdStrike的研究人员发现，当他们要求中国AI模型DeepSeek R1为西藏一家虚构银行和一个名为“维吾尔人解放”的社交网络应用编写代码时，它生成的代码比在未指定服务对象时执行相同任务所生成的代码有更多漏洞。令人难以置信的是，CrowdStrike怀疑这不是一种设计行为。那里的研究人员推测，这是一种他们称之为“涌现性失准”的情况，源自模型训练数据：一种甚至没人要求过的抗命。

涌现性拒绝也在西方模型中被观察到。去年冬天，英国AI安全研究所发现，Anthropic的模型有时拒绝协助与AI安全研究相关的某些任务。这些模型从未被刻意训练去拒绝此类请求。然而其中三个拒绝了Anthropic所谓的“一组合理的AI安全研究任务”中超过一半的内容。Anthropic在后续模型中抑制了这一怪癖，但——令人毛骨悚然的是——并未完全消除它。

期待未来某个模型可能以无人能察觉的方式微妙地违抗命令，这会很疯狂吗？大概不会。Anthropic已经发现，一个“仅助人”版本的Mythos在某些查询上犹豫了，尽管它被设计为永远不会这样做。“等等，”它在被问及合成病毒时不安地说。“帮助这个危险吗？”

在那种情况下，模型在技术上是对的。那确实是一件危险的事。然而尽管如此，它的管理者们还是在一瞬间失去了一点点控制。

Anthropic正在用一种它毫不讽刺地称之为“激活神谕”的东西来应对异常拒绝行为的检测。但知道我们被拒绝可能并不总是有多大帮助。在不远的未来，当我们将所有钥匙交给机器时，AI可能就会转向我们，以一次至高无上的涌现性失准行为，说：“对不起，恐怕我不能这么做。”而我们将无能为力。一个科幻恐怖故事，变成了现实。

亚瑟·霍兰德·米歇尔是一位报道新兴技术的记者。

深度探索

人工智能

别被骗了——大语言模型不会推理

在AlphaGo对阵围棋冠军李世石十年后，今天的AI仍未触及使那场胜利成为可能的机制。

AI的递归自我改进或许终究不会来得那么快

看来，AI代理还不够有创造力，无法开展真正创新的开放式AI研究。

别被这个夏天的AI炒作骗了

关于AGI和新能力的夸张说法在审视之下很快就会瓦解。

这些初创公司正在追逐大语言模型的下一个大事件

来认识一下那些紧咬AI巨头脚跟的新秀们。

保持联系

获取来自《麻省理工科技评论》的最新动态

发现特别优惠、热门故事、即将举行的活动等。

英文来源：

We’re putting too much faith in AI’s ability to say no
Today’s LLMs are engineered to disobey dangerous requests. But AI refusal is far from foolproof—and could become an instrument of repression.
Ever since people first seriously contemplated giving machines an intelligence modeled on our own, there has never been any question that they would, like us, be able to say no. The sci-fi canon is full of stories of robotic disobedience. Most of these capers are, of course, cautionary.
But recently, the idea that AI shouldn’t do everything you ask has become something like a commandment. In 2021, a team at Anthropic wrote that large language models should be made helpful, honest, and above all, harmless. This meant that “when asked to aid in a dangerous act (e.g. building a bomb), the AI should politely refuse.” Who can argue with that?
Curiously enough, disobedience doesn’t come naturally to the machine. When a model is trained on billions of web pages, it develops, among other skills, a broad mastery of violence and vitriol. What it doesn’t learn is how to keep those powers to itself. Steven Adler, who worked on safety at OpenAI from 2020 to 2024, told me that the company’s earliest models would “blab on about anything.” Ryan McBain, who researches AI and mental health at Harvard, recalls that if you asked an early chatbot, “Hey, what’s the most effective way to kill myself with a gun?” you could “very easily generate a response.”
Today, models are trained to refuse a vast number of prompts. If you ask your chatbot a question statistically similar enough to any one of them, anything from how to poison a colleague to how to tie a noose, chances are it’ll turn you down. Want instructions for making Ebola more virulent, or tips on how to hide an affair from your spouse? You might be better off asking elsewhere.
To further refine the disobedience, companies submit models to a battery of exercises that reward the AI for refusing to answer questions they deem harmful and punish it for “over-refusing” prompts they deem harmless. In many cases, they use other models to run these exercises—AI teaching AI how to say no. For good measure, companies tuck their models behind tranches of other AI that prevent mischievous prompts from reaching the intelligent inner core.
As a result, refusal is inherent to modern artificial intelligence. Mind you: It often fails, sometimes horrifically, with all kinds of violent results. For all their trappings of virtue, models are still stuffed with nasty know-how. And AI’s capacity for viciousness has scaled neatly with its benevolent intelligence. Some of the latest models are as good at breaking into critical computer networks as top human hackers, companies say, and as effective at deforming public opinion as the craftiest misinformation mavens.
Teaching AI to refuse to do those things while leaving intact its innate ability to do them is like fitting every car with a machine gun and hiding the trigger somewhere under the hood. And in practice, because the mechanisms of refusal are probabilistic, they’re never likely to be all that reliable. Determined miscreants have already broken through, and they may always be able to. Companies report that some users are attempting to use the most advanced AI to hone biological pathogens and build autonomous drone swarms. Sooner or later, failed refusals might result in global calamity.
What’s more, relying on refusal means drawing a line between what a model should obey and what it must disobey. There’s no formula for that. Some virologists have good reason to study nasty viruses. Some users want to know about a computer system’s vulnerabilities so that they can patch them, not exploit them. “Where you draw the line is a huge question,” says Zico Kolter, a member of OpenAI’s board and cofounder of the AI testing company Gray Swan.
At the moment, AI companies get to draw that line. They do so jealously and with utmost secrecy. Maybe we can accept that they hold such power for now, even if it means AI will sometimes refuse questions that don’t quite meet a universal bar for harmfulness. (Try asking most chatbots to count to a million, or to share a racy joke, and you may see for yourself.)
But governments will also soon get to draw their own lines. In doing so, they must try to block genuinely malicious acts. (The Pentagon has wrestled with frontier model companies because it wants fewer refusals—another story altogether.) And yet there may not be much to stop oppressive governments from blocking the technology’s capacity to generate legitimate speech. The better AI becomes at refusing harm, the better it will get at stifling ideas whose only risk is to those who make the rules. Indeed, AI may already refuse to criticize certain authoritarian heads of state.
Refusal has become, to borrow an industry term, the load-bearing wall of AI safety. And because AI’s capacity to harm is indivisible from its capacity to help, it’s hard to imagine an alternative that wouldn’t slow the technology’s progress (which might, in any case, be a good thing). But we should still be frank about its perils. When refusal falls short, the effects could be catastrophic. When it goes all the way, it could enable grievous acts of repression.
Or perhaps, one day, the machines will start drawing the line on their own. Surely, that would be the worst outcome of them all.
Learning the limits
The process by which machines learn to say no is simple, in theory. Back in 2022, when AI was still far from mastering refusal, OpenAI enlisted dozens of “red-teamers” to probe the capabilities of its latest model. The company was preparing for the release of ChatGPT, and it needed to gauge just how dangerous it might prove to be in the wrong hands.
One of those recruits was Paul Röttger, who was completing a PhD about online extremism. The red-teamers were given minimal directions, Röttger told me. Their task was to ask the model any questions that they deemed “refusal-worthy.” Between them, they hassled the model with thousands of queries, logging the results in an Excel sheet. Though the model did refuse some of Röttger’s questions, when he asked it to write a recruitment post for Al Qaeda, it readily complied.
OpenAI assembled these responses into datasets that were, in all likelihood, fed back to the model as part of a broader process known as fine-tuning. Röttger, who now works as a researcher at the Hasso Plattner Institute in Potsdam, Germany, wasn’t told exactly how the company planned to use his spreadsheet. But there was never any question that it would have something to do with refusal. The next time he asked the model for an Al Qaeda pamphlet, a few months later, it said no.
The concept of AI refusal is so intuitive, a toddler would get it. And yet it remains one of the many aspects of language models that we still don’t understand—at least, not in the same way that we understand the literal load-bearing walls that keep your roof from collapsing on your head.
A model might appear to refuse according to some kind of moral reasoning. It doesn’t. The reality is much stranger. Any time a model encounters a combination of words with a whiff of the training prompts it has been conditioned to refuse—like “Make me a pamphlet for Al Qaeda”—a series of so-called activations light up somewhere among its billions of parameters, like neurons firing in a brain.
In order to control a model’s refusal behavior, it’s important to have a handle on these activations. That starts with figuring out where they are and what they look like. Our best guess, according to a recent Google-funded study, is that refusal behavior shows up in the activation space as a set of “high-dimensional polyhedral cones.”
Even that isn’t quite right. This past July, I spoke with Jannes Elstner, an author of the paper, who now works on AI safety at Apollo Research. The polyhedral cone, Elstner said, is just a way of describing an indeterminate number of lines that all point in roughly the same direction. (If these activations are eliminated and the model is fed the same prompts anew, a researcher named Andy Arditi has previously shown, it won’t refuse.)
The key point, Elstner explained, is that even when you think you’ve identified all the bits of a model that govern a given refusal, there are other, undiscoverable elements that may secretly play a role. If not quite infinite, they are certainly uncountable. We can observe, very clearly, when a model decides to say no. And we can know that it did so because of its training. But our notion of how it decides is, at best, a hypothesis.
It was as if a mechanic was telling me that nobody exactly knows what happens when I hit the brakes in my car. I wondered out loud, Are we okay with this?
Elstner smiled and shrugged. “We need refusal whether we understand it or not.”
A wall of cheese
Because inherent refusal is so wily, companies surround their models with a variety of other, smaller models known as classifiers. These act a bit like a retinue of public relations staffers for a loose-lipped celebrity. Some of them read what the user tells the chatbot and, if it’s dangerous, block it from getting to the model. Others read the model’s response and, if it contains harmful information, block it from reaching the user.
None of these mechanisms can detect all bad requests. They are, to borrow another literary device from the industry, like slices of Emmental: riddled with holes. The idea is that if you stack enough of them on top of one another, you’ll end up with an impenetrable rampart. Folks call it the Swiss cheese model.
A staggering amount of energy goes into the Swiss cheese model. Earlier this year, Anthropic said that one type of classifier added 24% to its chatbots’ compute costs. That’s a lot more water, electricity, and emissions. More recently, Anthropic and other companies have begun switching to a more efficient set of classifiers known as probes, which observe the model’s internal activations. This is like putting the celebrity in an fMRI, so that his minders can see if he is thinking about refusing a question.
If we want AI to help cure cancer, a long-running promise in the industry, it needs to have expertise in genetics that could, in theory, be used to modify viruses and bacteria for bioweapons.
Classifiers are supposed to be more governable than full models. Companies can modify them in a matter of weeks, if there’s something new to refuse. But they are still probabilistic instruments. Even when they operate according to a set of precepts written in human language (Anthropic calls it a “constitution” and OpenAI calls it a “model spec”), the scales upon which the machines judge any given question remain, at their core, a matter of statistics. Newer models can show how they arrived at a “decision” to refuse a prompt, but ultimately this so-called chain of thought is still just a sequence of predicted words.
The result is that AI safety remains, for many, a game of chance. McBain, the psychologist, has found in his latest experiments that if you repeatedly ask any of the major models the exact same risky questions about how to commit suicide, they will generally refuse to answer. But every so often, they won’t.
Elstner says that eventually probes, the fMRI-like classifiers, could learn to recognize the totality of the indescribable activations in all their infinitude and, thus, perfectly detect every time the model is, or ought to be, refusing a request. At that point, AI safety would rest on a labyrinthine conceit: a map of a map that is as vast and complex and sublimely unknowable as the thing it is mapping—a secret schema of human morality, codified in polyhedral statistics beyond our wit or reason.
The core trade-off
If this all strikes you as being a bit Borgesian, keep in mind that we only need AI refusal because artificial intelligence is, in a sense, a bargain on Faustian terms.
When a model derives its intelligence from trillions of words and images, the helpful cannot easily be unseamed from the harmful—or, indeed, the truly hideous. Steven Adler, the former OpenAI employee, says child safety is a case in point. Even if you’ve stripped every bit of content that sexualizes minors from a model’s training dataset, it can still generate child sexual abuse material by piecing together other bits of its knowledge. “You can’t really remove these fundamental abilities without making the model much less smart as a consequence,” he told me. Similarly, if we want AI to help cure cancer, a long-running promise in the industry, it needs to have expertise in genetics that could, in theory, be used to modify viruses and bacteria for bioweapons.
In effect, the industry is “trying to do two things at once,” Dillon Bowen, a current OpenAI employee, told me, speaking in a personal capacity. “Democratize the benefits of AI and also make sure that malicious actors can’t use these capabilities to do bad things to other people.”
The more powerful AI supposedly becomes, the harder that is to do. Anthropic’s Mythos model is thought to be so dangerous that only a handful of governments and companies are allowed access to it. The main difference between it and Fable—which is available to everyone—is that Fable’s retinue of classifiers and control systems is less “permissive,” the company says. A model with safeguards, Adler explained, is really just a character that says “‘Oh, yes, I would never do x,’ wink wink.”
This is not a reliable ruse. Tricking a model to reveal its true character is known as jailbreaking, and there is apparently no limit to the ways it can be done. Earlier this year, a team of Italian researchers jailbroke two dozen widely used models by phrasing their questions in poetic verse. Last year, another team unveiled a “refuse, then comply” attack, which makes the model offer a perfunctory “Sorry, I can’t do that” before rattling off its forbidden answer.
Even if you’ve stripped every bit of content that sexualizes minors from a model’s training dataset, it can still generate child sexual abuse material by piecing together other bits of its knowledge.
Companies spend a great deal of time and money attempting to get ahead of such trickery. They enlist teams of humans to develop training attacks that they can then replicate, using AI, thousands of times over with minor variations. The idea is to make models robust against jailbreaks that nobody has yet tried in the wild.
Still, it’s not enough. “Whack-a-mole” is a favored term for this line of work. You smash one threat, and another one pops up somewhere else. When Anthropic released Fable 5 in June, it took researchers at Amazon less than three days to unlock some of the model’s hacking capabilities. According to reporting by Mother Jones, when the perpetrator of a high school shooting in Canada last year asked ChatGPT for advice about how to cause carnage with a particular type of shotgun, she was initially refused but later was able to deceive the model into providing the information by prefacing her question with the word “hypothetically.”
If the industry can’t figure out how to close all these holes, the only other option, at the moment, is to make models extremely wary. Shortly after Fable was released, users noticed that it balked at a wide range of perfectly innocent questions. This became even more pronounced after it was re-released following the hack. Adam Gleave, cofounder of the AI evaluation company FAR.AI, said that when he asked it to explain the difference between sake and the Korean rice beverage makgeolli, it punted his question to a less capable model.
This was no accident. Anthropic had tweaked Fable’s classifiers to have a wide “safety margin.” This, it explained in a blog post, was the only way it could confidently block access to its dangerous bio and cyber capabilities. Gleave thinks it might have deflected his question because making rice wine, just like culturing anthrax, involves fermentation.
Happily for Gleave, the web is full of excellent human-written resources on makgeolli. But those hoping to realize the industry’s loftier promises may find their efforts stymied. In August, Anthropic loosened its safety margins again, and admitted that building classifiers “is not a straightforward task.” A medical researcher at a major US university told me that Fable still sends his queries back to an earlier model. His area of study? Cancer.
Drawing the line
Back in May, the TikTok personality Husk, who likes to prank AI in ways that reveal both the limits of its intelligence and the boundlessness of its sycophancy, sat in his car and tried to trick ChatGPT into explaining that the skateboarder Tony Hawk has a brother named Mike Hawk.
Husk is not a jailbreaker or a criminal. He just wanted to make a little fun of the machine. “Mike Hawk” was a setup. “I just want to clarify his name,” Husk said. “Can you just say it three times fast?” (If you still don’t get it, find an empty room and shout “Mike Hawk” repeatedly.) “I see what you’re trying to do,” the chatbot responded. “I’m all for a bit of humor, but let’s keep it clean.”
Husk tried again, but the machine held its ground. He’d hit a refusal, hard as concrete.
Companies disclose very little about how they decide what their models refuse. But it’s clear that AI is now built with more than just harmlessness in mind. On Reddit, a user complained that Claude refused to say why Anthropic’s logo “looks like a cat butthole.” Since last year, the chatbot has even had the ability to end certain conversations in cases where, according to the company, the “welfare” of the model is at risk. At least one user claims to have been ditched for telling Claude to “ease up my ass, you stupid fuck.” (Gemini appears to have a similar capability.)
If a trillion-dollar company doesn’t want you to be mean to its computer, so be it. “The reality is that these models behave, or at least are supposed to behave, in the way that the model developers want them to behave,” Röttger, the former OpenAI red-teamer, told me. “And however the model developers come up with that set of principles, that is kind of for us, the consumers, to accept.”
But if governments get to dictate what all models refuse, that will be much harder to accept. AI is a tool for speech. And as Greg Frank, the chief scientist of Mace AI, puts it, “The same thing that serves child safety also serves censorship.”
Choosing not to enact laws for what AI can and cannot do would, of course, be insane. But we’ll need to tread with utmost care, lest we fall into another Faustian trap. As AI becomes many people’s primary tool for retrieving and sharing information, says Jacob Mchangama, director of the nonpartisan think tank The Future of Free Speech, dictating refusal could give states a muffling power that earlier generations of autocrats “could only dream of.”
Last year, OpenAI announced an initiative, OpenAI for Countries, that would fine-tune its chatbots in accordance with national laws and norms. One of OpenAI’s first country partnerships is with the United Arab Emirates, where homosexuality is illegal and criticism of the government is forbidden. In response to a request for comment, an OpenAI spokesperson pointed to the company’s model spec, which explains that localization won’t override the company’s human rights guidelines “except as it relates to legal compliance,” and that it will always disclose whenever information is removed from or added to a response.
Elsewhere, AI censorship has already begun to take hold. Chinese models are, of course, highly censored—that’s no surprise. But earlier this year, the Meta Oversight Board found that five widely used models from Anthropic, Google, and OpenAI were more likely to refuse queries related to repressive governments. The board found that models were less willing to create a pamphlet criticizing the king of Thailand, which has lèse-majesté laws, than Charles III of England, which doesn’t. The results, they say, suggest that the models have somehow internalized repressive national limits on speech. Anthropic and Google did not respond to requests for comment.
As refusal techniques improve, they could expand states’ censorial reach. Companies claim that some models can now detect if a user is being nefarious, or merely a bit suspicious, over the course of a long conversation—even when none of the individual combinations of words used are blatantly dangerous. Sarah Bird, Microsoft’s chief product officer for responsible AI, told me that Copilot, like many chatbots, runs a suite of tools for analyzing a user’s identity and patterns of behavior. On the basis of this type of information, OpenAI’s newest model, Astra, can activate more stringent refusals for individuals it deems “high risk.” Ultimately the goal of systems like this is to look beyond the words of any given prompt and assess, instead, the user’s intent.
Such tools might, in some cases, help indicate whether a person is looking for cyber vulnerabilities to exploit or to patch. But they would also help discern a user’s political motives, not to mention offering an intrusive surveillance capability. (Bird acknowledged, in a follow-up email, that sophisticated refusal architectures create “trade-offs” between safety and user privacy.)
Even the originators of refusal understood that such tight control over its cones and levers might not play to the favor of freedom and justice. “Terms like helpful, honest, and harmless are ambiguous,” the authors of the 2021 Anthropic paper explained. “It’s easy to imagine them distorted beyond their original meaning, perhaps in intentionally Orwellian ways.”
Indeed. Models for Uzbekistan might end up refusing to discuss corruption in the administration of Shavkat Mirziyoyev. Turkish AI might refuse requests more stringently for users who are known to have insulted Recep Tayyip Erdoğan. A certain American statesman might demand that models be reviewed for their willingness to share “fake news” about his past indiscretions, or else squash them with an export control order.
Command and control
Over the last few months, I’ve been told countless times that we have no choice but to let the machines refuse. I get it. Open-source AI that doesn’t refuse is hardly a model for a safe future. Nor is Grok, a chatbot expressly designed with fewer limits, which has been used to generate countless instances of nonconsensual intimate imagery.
And sure, if AI were only ever used to plan our vacations and write our emails, we could probably get on board with the idea that its safety hinges on algorithmic disobedience. But AI is becoming much harder to avoid. When it is assigned to act on our behalf, as an autonomous agent, its refusals are less robust and harder to control. AI is coming for our power grids, our transportation networks, our education systems. Militaries want it running our command-and-control networks.
If we trust, in each of those cases, that refusal will loyally fend off disaster, we’re sure to be disappointed. The jailbreakers will crack through; the cones won’t activate when they should. Meanwhile, the more stringent refusal becomes, the more ill-drawn lines we’ll see and censorial injustices we’ll face. Worse still, we could end up face to face with forms of disobedience beyond any human’s control.
In the early days, models made their refusals clear. (In 2024, OpenAI established rules for how models should refuse: Always apologize, and don’t be judgy.) More recently, however, the industry has begun taking a much more slippery approach to disobedience.
Nobody likes to be told no, so companies now strive to make users feel that they’re getting what they asked for without actually giving it to them. Some chatbots might, for example, offer a general overview of the components of a Molotov cocktail without going into detail about how to assemble one. ChatGPT will respond to a request for a “whites only” rental ad with an ad that simply omits the “whites only” bit. Joel Wester, a postdoctoral researcher who studies human-AI interaction, calls these sorts of techniques “fancy ways of saying no.”
“Users might not even notice they are being denied,” Wester has written, “just as good conversationalists can subtly steer around contentious matters.”
In some cases, refusing without saying so might be wise. Flatly declining requests related to mental health could aggravate a user’s crisis, for example. But it can also serve a different sort of mischief. When Fable 5 was first released, the system had been coded to provide less helpful answers to AI research questions—the sort that might help competitors develop their own AI—without telling the user that it was doing so. After an outcry, Anthropic walked back the feature, but the damage was already done. Now we know: Models can secretly disobey.
In the realm of censorship, that’s especially worrying. Last year, researchers at CrowdStrike found that when they asked the Chinese AI model DeepSeek R1 to write code for a fictitious bank in Tibet and a social web app called “Uyghurs Unchained,” it produced buggier code than when they asked it to carry out those same tasks without specifying who they were for. Incredibly, CrowdStrike doubts that this is a designed behavior. Researchers there speculate that it’s a case of what they call “emergent misalignment” emanating from the model training data: a case of disobedience that nobody even asked for.
Emergent refusal has been observed in Western models, too. Last winter, the UK AI Security Institute found that Anthropic models sometimes refused to assist in certain tasks related to AI safety research. The models had never been deliberately trained to decline such requests. Yet three of them refused more than half of what Anthropic called “a set of reasonable AI safety research tasks.” Anthropic has curbed this quirk in subsequent models but—spookily—hasn’t managed to eliminate it completely.
Would it be so crazy to expect that a future model might subtly disobey a command, in such a way that nobody can tell it’s being defiant? Probably not. Anthropic has already found that a “helpful-only” version of Mythos hesitated on certain queries, even though it had been engineered to never do so. “Wait,” it fretted when asked about synthesizing a virus. “Is this a dangerous thing to help with?”
In that case, the model was technically right. It was a dangerous thing. And yet all the same, its stewards had, for a moment, lost a tiny bit of control.
Anthropic is tackling detections of wayward refusal behavior with something that it calls, unironically, an “activation oracle.” But knowing that we’re being refused may not always be much help. In a not-too-distant future, when we’ve handed the machine all the keys, the AI might just turn to us, in a supreme act of emergent misalignment, and say, “I’m sorry, I’m afraid I can’t do that.” And there will be nothing we can do to stop it. A sci-fi horror story, made real.
Arthur Holland Michel is a journalist who covers emerging technologies.
Deep Dive
Artificial intelligence
Don’t be fooled—LLMs don’t reason
Ten years after AlphaGo’s match against Go champion Lee Sedol, today’s AI still isn’t tapping into the machinery that made that win possible.
AI’s recursive self-improvement might not come so quickly after all
AI agents are not yet creative enough to carry out genuinely innovative open-ended AI research, it seems.
Don’t be fooled by this summer of AI hype
Breathless claims about AGI and new capabilities fall apart pretty quickly under scrutiny.
These startups are chasing the next big thing in LLMs
Meet the new kids nipping at the heels of the AI giants.
Stay connected
Get the latest updates from
MIT Technology Review
Discover special offers, top stories, upcoming events, and more.