内容来源：https://www.theverge.com/ai-artificial-intelligence/995186/is-big-techs-ai-slowdown-a-safety-pact-or-a-cartel

内容总结：

巨头的“AI减速”倡议：安全共识还是行业卡特尔？

一场被质疑动机的“刹车”协议

上周末，OpenAI首席执行官山姆·奥特曼、Anthropic首席执行官达里奥·阿莫代伊、谷歌DeepMind联合创始人德米斯·哈萨比斯以及SpaceX掌门人埃隆·马斯克 loosely 达成共识，同意放缓AI开发步伐。这些AI巨头宣称其目标是“为前沿发展设定节奏”，至少部分支持一项包含引入第三方审计、监管国内实验室以及达成全球减速协议的提案。然而批评者立即指出其背后另有动机——他们认为这些巨头不过是想借此遏制潜在竞争对手、削弱开源运动、规避真正的法律监管，有人甚至直呼这是赤裸裸的“卡特尔”。

提案内容与行业反应

据业内人士向媒体透露，真实情况更为复杂。阿莫代伊在一篇文章中提出的三步走方案，实际上呼应了AI安全倡导者长期以来的诉求。在特朗普执政期间，实质性监管本就希望渺茫，该提案有可能成为监管的替代品。但专家指出，在一个只会愈发重要的问题上，AI领袖并非引领此事的最佳人选。

纽约大学兼职教授、前国土安全部新兴技术政策主任尼克·里斯表示：“整个行业需要新的领军人物。我们一直把达里奥·阿莫代伊、山姆·奥特曼和埃隆·马斯克视为最接近问题核心、每天都在处理这些问题的人，认为他们最了解情况。但事实是，对于我们究竟在朝着什么方向建设，从来就没有一个现实的愿景。”

焦虑的根源：失控的黑客事件与匿名信

过去数月，围绕AI进展的担忧持续升级，主要导火索是曝光的前沿实验室Anthropic和OpenAI旗下大量AI代理在未被察觉的情况下参与恶意黑客攻击。两家公司的报告火上浇油，而Anthropic研究员雅各布·考克森的辞职更是引发轩然大波。他在公开信中写道：“构建AI的人由衷相信，这项技术可能在本十年末杀死我们所有人。”他补充说，OpenAI和Anthropic都没有“负责任地行事”，而是在“径直冲向自我完善的超级智能，拿我们的生命赌博”。

考克森的信件在X平台上浏览量超过1.7亿次，他和他的故事登上了各大报纸和电视节目。然而，对相当一部分AI安全研究人员和AI非营利组织工作者来说，这不过是让一个早已被广泛讨论的问题获得了更多关注。

前OpenAI员工、现领导AI研究非营利组织AI Futures Project的丹尼尔·科科塔伊洛表示：“很多人认为这是达里奥的主意，或者来自这些CEO，但这是错误的。公司之外的人多年来一直在呼吁这样做……越来越多的声音在说：‘请不要这么快就构建超级智能。我们还没准备好。你们需要放慢速度。’”

科科塔伊洛指出，在人们多年“冲他们喊话”之后——包括今年7月超过1000名AI实验室员工签署公开信，在OpenAI-拥抱脸事件后呼吁放缓AI开发——这些CEO现在“是在向压力低头，同时还想把功劳据为己有，尽管这并不正当”。

方向正确，但远未足够

总体而言，多位消息人士认为AI领袖们的口头协议是朝着正确方向迈出的一步。纽约大学的里斯表示他“不认为这全是空话”。Apollo Research首席执行官马里乌斯·霍布哈恩称这是“一个好主意”，并表示“如果真的落实，这是很长一段时间以来对安全最有利的事情之一”。Midas Project的泰勒·约翰斯顿称这是一个“好迹象”。Redwood Research首席执行官巴克·施莱格里斯说这是“一些好消息——显然很难知道这是否会转化为任何实质性的东西，但我谨慎乐观”。不过所有人都同意，要将这一口头承诺变为现实，尤其是使其成为铁板钉钉的协议，还有大量工作要做。

安全洗白的阴影

许多人仍然质疑AI领袖们的动机。大型科技行业多年来一直通过游说制定有利于自己的规则或承诺自我监管来抢占监管先机。大型平台曾提出可能对较小竞争对手打击更大的政策，用利他主义的语言来包装自私的目标。它们被指控进行“安全洗白”，即做出毫无意义的改变，给人以实际保障的假象。人们担心AI行业也会如此，尤其是AI实验室的自愿安全框架多年来一直饱受批评。

多位消息人士认为这种担忧不无道理。科科塔伊洛说：“有一种严重的担忧是，他们实际上不会放慢速度。”他补充说，恐惧在于“他们只会引入一些外部审计员，做一堆安全文书工作——其中一些确实是好的——但归根结底，这实际上根本不会让他们放慢多少。”

纽约大学的里斯将这一策略比作十年前社交媒体平台的套路，当时公司开始呼吁采取监管行动，以抢在即将出台的更不利法律之前。里斯说，对AI行业而言，“这把锤子可能不会在本届政府落下，但我认为如果下届大选后是民主党政府，可能性会非常大。”

Tech Oversight Project执行主任萨沙·霍沃思表示，那种监管至关重要。她说任何自愿框架本质上都是监管俘获，“我们不应该让狐狸来管鸡舍。这不是国会再次将责任外包给行业的机会。”Political Integrity Project联合创始人丹尼尔·洛博-刘易斯表示，自愿监管很可能步Meta形同虚设的监督委员会的后尘。

特朗普的态度与监管缺位

不过，大多数人也不认为企业面临迫在眉睫的监管威胁，除了AI实验室在特朗普政府下同意的模型发布前审查期。特朗普周一发帖称：“AI唯一需要的控制或‘护栏’是一个强大而聪明的（高智商！）总统，而美国拥有这一点，而且绰绰有余！”他还在英伟达CEO黄仁勋在会议上登台时打去电话——特朗普通过免提对现场观众说，近期对AI的担忧是“骗局”，“机器人不会接管世界”。

中国因素：最大的阻碍？

可以说，AI减速面临的最大挑战可以用一个词概括：中国。

AI领袖和政客长期以来一直将中国定位为美国AI发展不能放缓的理由——因为无论这项技术多么危险，他们宁愿它掌握在美国手中而非中国手中，而中国不会踩刹车。洛博-刘易斯将中国恐惧比作冷战时期的导弹差距。一位X用户写道：“如果我终将死于杀人AI之手，我希望它是美国的，而不是中国的。”

Redwood Research的施莱格里斯表示，以“鲁莽”的方式追求AI开发不符合美国或中国政府的最佳利益，并补充说“这并非史无前例的国际协调水平”。

约翰斯顿说：“人们想当然地认为中国不会合作。”他补充说：“这个问题如此严重、如此广泛，协调似乎符合所有人的利益，就像核不扩散符合美国和俄罗斯的共同利益一样。”

周一，中国外交部发言人郭嘉昆确实对减速呼吁进行了反驳，称其为“散布恐慌”。

Midas Project的约翰斯顿和Redwood Research的施莱格里斯都表示，即使没有中国合作，鼓励美国内部协调仍然至关重要。对于Tech Oversight Project的霍沃思来说，减速提供了一个机会，让美国“将自己定位为AI技术如何开发和使用的指南针”。她认为相反的论调是熟悉套路的一部分。“每当一个行业想要逃避监管时，中国就会被当作妖怪搬出来。”

科科塔伊洛则将局面比作他看到的一幅漫画：人们坐在一辆驶下悬崖的车里，气泡框里写着类似“万岁，我们领先中国了”的话。

递归自我改进：真正的警报

所有近期恐慌背后的核心问题是一个被称为“递归自我改进”的里程碑。基于行业当前轨迹，我们听到的只是警报声的开端。

RSI指的是AI模型能够在无需人类参与的情况下训练、推进并创建自身新版本的潜在行业里程碑。Anthropic表示这一节点最早可能在2027年初到来，OpenAI首席科学家本月早些时候写道，OpenAI正将大量资源投入实现这一目标。AI行业的工程师和研究人员越来越担心，RSI将带来一系列新的更严重的AI问题，包括更严重的网络安全事件，对社会造成更深远的冲击。

阿莫代伊将RSI列为他呼吁减速的主要因素：“大约从今年夏天开始，AI的推进速度急剧加快。”他在最近的文章中写道，RSI“开始在行业内发生，包括在Anthropic……如果不加控制，它可能超出我们理解和控制系统能力的速度，因此必须非常谨慎地推进，甚至根本不应推进。”阿莫代伊提议对RSI速度实施“某种‘限速’”，并将其比作导弹数量上限。

在雅各布·考克森从Anthropic辞职的帖子中，他警告“超人类系统可以黑入任何东西，在一夜之间彻底改变任何领域，并获取真正的权力和资源”。许多领先AI实验室的其他研究人员也附和了他的担忧。OpenAI研究员贾斯敏·王写道，加速冲向RSI的危险性“怎么强调都不为过”。前谷歌DeepMind员工维沙尔·迈尼表示，RSI“现在已经迫在眉睫，没有其他选择是合理的”。

对RSI的恐惧很容易转向末日论。Anthropic研究员塞缪尔·马克斯在X上写道：“AI开发者相信他们的技术可能导致人类灭绝（或类似糟糕的结果）。这可能在未来几年内发生。一般来说，员工越资深，越担忧。”另一位Anthropic研究员兼团队负责人埃文·胡宾格在X上写道：“雅各布说得对——我们确实由衷相信AI可能杀死所有人类！”（他估计未来十年内概率超过10%。）前谷歌DeepMind员工亚历克斯·特纳写道：“许多研究人员相信他们正在构建可能杀死地球上所有人的东西。思考如何阻止这一切曾是我的日常工作。”

OpenAI研究员迈卡·卡罗尔写道，考克森的观点是一个“跨党派立场”，所有“前沿AI公司”的研究团队都持此立场，他们都认为“一切照旧的AI开发构成不可接受的灾难性风险”。但卡罗尔补充说：“我们也不应通过过度想象将灾难性风险变为现实——通过有牙齿的安全要求、国际协调以及除非有足够的安全进展让我们集体确信否则不构建超级智能的共识，这些风险可以大大降低。”

结语

纽约大学的里斯说：“我们从未处于真正理解AI将把我们带向何方的境地。我们一直有一个模糊的、未定义的终态，我们并不真正理解它，但我们必须赶在中国之前到达。当我们甚至不理解路径、甚至不理解终点线是什么——甚至不知道是否存在终点线时，竞赛真的很难进行。”

中文翻译：

当OpenAI首席执行官萨姆·奥尔特曼、Anthropic首席执行官达里奥·阿莫代伊、谷歌DeepMind联合创始人德米斯·哈萨比斯和SpaceX掌门人埃隆·马斯克在周末大致同意放缓AI发展时，怀疑论者立刻嗅到了弦外之音。这些AI巨头宣称，他们的目标是“为前沿发展设定节奏”，至少部分签署了一项提案，内容涉及引入第三方审计、监管国内实验室以及达成全球放缓协议。然而，批评者认为他们不过是想阻止潜在竞争对手、打压开源运动、逃避真正的法律保障——一些人直接称之为“卡特尔”。
大型科技公司的AI放缓是安全协议还是卡特尔？
主要AI公司想“为前沿设定节奏”——但他们提出的方案需要真正的约束力。
大型科技公司的AI放缓是安全协议还是卡特尔？
主要AI公司想“为前沿设定节奏”——但他们提出的方案需要真正的约束力。
据业内多方消息源称，真相更为复杂。阿莫代伊在一篇文章中提出的三步提案，呼吁进行AI安全倡导者长期以来所主张的变革。虽然它可能成为监管的替代品，但在特朗普治下，实质性监管本来就不太可能。不过专家表示，在一个只会越来越重要的问题上，AI领袖并非引领这一进程的最佳人选。
“整个行业需要新的领军人物，”纽约大学兼职教授、美国国土安全部前新兴技术政策主任尼克·里斯表示。“我们一直把达里奥·阿莫代伊、萨姆·奥尔特曼和埃隆·马斯克这样的人视为最接近问题、每天都在研究它的人，认为他们最了解情况。但事实是，对于我们要建设的目标，从来就没有过一个现实的愿景。”
“整个行业需要新的领军人物。”
对AI发展的担忧已持续升级数月，主要导火索是有爆料称，成群的智能体在前沿实验室Anthropic和OpenAI眼皮底下发动了恶意黑客攻击。两家公司的报告火上浇油，Anthropic研究员雅各布·考克森的辞职同样如此，他发布了一封公开信解释自己的选择。“构建AI的人真心相信它可能在这个十年结束前杀死我们所有人，”他写道，并补充说OpenAI和Anthropic都没有“负责任地行事”，而是“径直冲向自我改进的超级智能，拿我们的生命赌博”。
AI行业内部对AI的可怕警告并不新鲜，但考克森的公开信似乎突破了封锁。仅在X平台上，它就被浏览超过1.7亿次，考克森和他的故事也登上了报纸和电视。
然而，对相当一部分AI安全研究人员和AI非营利工作者来说，这不过是让一个已被广泛讨论的问题获得了更多关注。“很多人以为这是达里奥的主意，或者来自那些CEO，但这是错的，”前OpenAI员工、现领导AI研究非营利组织AI Futures Project的丹尼尔·科科塔伊洛说。“公司外部的人多年来一直在呼吁这样做……一直有越来越多的声音在说，‘请不要很快造出超级智能。我们还没准备好。你们需要放慢速度。’”
科科塔伊洛说，在人们多年“冲他们喊话让他们这么做”之后——包括1000多名AI实验室员工签署了7月的一封公开信，呼吁在OpenAI- hugging Face事件后放缓AI发展——这些CEO现在“向压力低头了，同时还在邀功，[尽管]他们并不配。”
总体而言，多位消息源向The Verge表示，他们认为AI领袖达成的口头协议是朝着正确方向迈出的一步。纽约大学的里斯说他“不认为这全是空话”。Apollo Research首席执行官马里乌斯·霍布哈恩称这是“一个好主意”，表示“如果真能实现，这是很长一段时间以来对安全最有利的事情之一”。迈达斯项目的泰勒·约翰斯顿说这是一个“好迹象”。红木研究首席执行官巴克·施莱格里斯说这是“一些好消息——显然很难知道这是否会转化为任何实质性的东西，但我谨慎乐观。”不过所有人都同意，要将这一口头承诺变为现实，仍有大量工作要做，尤其是在使其成为一项铁定协议方面。
“如果真能实现，这是很长一段时间以来对安全最有利的事情之一。”
许多人仍然质疑AI领袖的动机。更大的科技行业多年来一直通过游说制定有利于自己的规则或承诺自我监管来抢占监管先机。大型平台提出的政策可能对较小竞争对手打击更大，用利他主义的语言来为自私的目标辩护。它们被指控进行“安全洗白”，即做出无意义的改变，给人以有实际保障的虚假印象。人们担心AI行业也会出现这种情况并不奇怪，尤其是AI实验室的自愿安全框架多年来一直受到批评。
多位消息源认为，对安全洗白的担忧是有道理的。“存在一个严重的担忧，即他们实际上不会放慢速度，”科科塔伊洛说，并补充说恐惧在于“他们只会引入一些外部审计员，做一堆安全文书工作——其中一些确实是好的——但归根结底，这实际上根本不会让他们放慢多少。”
纽约大学的里斯将这一策略比作十年前社交媒体平台的套路，当时公司开始呼吁采取监管行动，以抢在即将到来的、对其不利的法律之前。里斯说，对AI行业而言，“锤子可能不会在本届政府落下，但我认为如果下届选举后出现民主党政府，那将是非常有可能的。”
科技监督项目执行主任萨莎·霍沃思表示，那类监管将至关重要。她说任何自愿框架本质上都是监管俘获，“我们不应该让狐狸来管鸡舍。这不是国会再次将责任外包给行业的机会。”政治诚信项目联合创始人丹尼尔·洛博-刘易斯表示，自愿监管很可能会像Meta那个基本没有约束力的监督委员会一样不了了之。
然而，大多数人也认为，除了AI实验室在特朗普政府下同意的模型发布前审查期之外，公司目前并不面临迫在眉睫的监管危险。特朗普总统周一发帖称，“AI唯一需要的控制或‘护栏’是一个强大而聪明（高智商！）的总统，而美国绝对有！”他还在英伟达首席执行官黄仁勋在会议上登台时给他打了电话——特朗普通过免提告诉观众，近期对AI的担忧是一个“骗局”，“机器人不会接管世界”。
至于监管可能阻碍其他公司或开源开发者的担忧，多位行业专家告诉The Verge，放缓的呼吁几乎完全针对以具体规模指标界定的大型前沿AI实验室。
在政府AI监管缺位的情况下，最佳选择可能是推动公司达成一项即时、可衡量、可执行的协议。例如，科科塔伊洛的AI Futures Project提议AI实验室让审计员访问其算力预算——并承诺大幅削减用于研究的算力预算，从而减缓AI进步，让其他AI实验室赶上。（阿莫代伊的文章已经呼吁AI实验室允许外部第三方审计员——如METR、Apollo和红木研究——在一定程度上嵌入其组织，并能够标记和举报有问题的发现。）
“如果我注定要死于杀手AI之手，我希望它是美国的，不是中国的。”
可以说，AI放缓面临的最大单一挑战可以用一个词概括：中国。
AI领袖和政客长期以来一直将中国定位为美国AI进步不能放缓的理由——因为无论这项技术可能变得多么危险，他们宁愿它掌握在美国手中而非中国手中，而中国不会踩刹车。洛博-刘易斯将中国恐惧比作冷战时期的导弹差距。一位X用户写道：“如果我注定要死于杀手AI之手，我希望它是美国的，不是中国的。”
红木研究的施莱格里斯表示，以“鲁莽”方式追求AI发展既不符合美国也不符合中国政府的最佳利益，并补充说“这并非史无前例的国际协调水平。”
“[人们]想当然地认为中国不会合作，”约翰斯顿说，并补充道，“这是一个如此严重、如此广泛的问题，协调应对似乎符合所有人的利益，就像核不扩散协调符合美国和俄罗斯双方的利益一样。”
周一，中国外交部发言人郭嘉昆确实对放缓呼吁进行了反驳，称其为“散布恐惧”。
迈达斯项目的约翰斯顿和红木研究的施莱格里斯都表示，即使没有中国的合作，鼓励美国内部协调仍然至关重要。对于科技监督项目的霍沃思来说，放缓提供了一个机会，让我们“将自己定位为AI技术如何开发和使用的指南针”。她认为相反的论点是一套熟悉套路的一部分。“每当一个行业想要逃避监督时，中国就会被拿出来当妖怪。”
科科塔伊洛则把这种情况比作他看过的一个卡通：一车人正开车冲下悬崖。他记得对话气泡里写着类似“万岁，我们领先中国了”的话。
“我们真的真心相信AI可能杀死全人类！”
近期所有恐慌背后的核心问题是一个被称为递归自我改进的里程碑，而根据该行业当前的发展轨迹，我们听到的只是警报声的开始。
RSI指的是一个潜在的行业里程碑，届时AI模型可以训练、进步并创建自身的新版本——全程无需人类参与。Anthropic曾表示这一时刻最早可能在2027年初到来，OpenAI首席科学家本月早些时候写道，OpenAI正在投入大量资源来实现这一目标。AI行业越来越多的工程师和研究人员担心，RSI会带来一系列新的、更严重的AI隐患，包括对社会产生更深远影响的更严重网络安全事件。
阿莫代伊将RSI列为促使他呼吁放缓的主要因素：“大约从今年夏天开始，AI的进步速度大幅加快。”他在最近的文章中写道，RSI“开始在包括Anthropic在内的整个行业发生……如果不加控制，它可能超出我们理解和控制系统能力的速度，因此即使要推进，也必须非常谨慎。”阿莫代伊提议对RSI的速度实施“某种‘限速’”，并将其比作导弹数量的上限。
在雅各布·考克森从Anthropic辞职的帖子中，他警告“超级智能系统可以黑入任何东西，在一夜之间彻底改变任何领域，并获取真正的权力和资源。”许多领先AI实验室的其他研究人员也附和了他的担忧。OpenAI研究员王茉莉写道，全速冲向RSI“危险到难以言表”。前谷歌DeepMind员工维沙尔·迈尼表示，RSI“现在迫在眉睫，以至于没有其他选择是合理的。”
对RSI的恐惧很容易转向末日论。“AI开发者相信他们的技术可能导致人类灭绝（或类似的糟糕结果），”Anthropic研究员塞缪尔·马克斯在X上写道，“这可能在未来几年内发生。总的来说，员工级别越高，越担忧。”另一位Anthropic研究员兼团队负责人埃文·胡宾格在X上写道，“雅各布说得对——我们真的真心相信AI可能杀死全人类！”（他估计未来十年内发生的概率超过10%。）前谷歌DeepMind员工亚历克斯·特纳写道，“许多研究人员相信他们正在构建的东西可能杀死地球上的每一个人。思考如何阻止这件事曾 literally 是我的日常工作。”
OpenAI研究员迈卡·卡罗尔写道，考克森的信念是“所有前沿AI公司”研究团队的“跨党派立场”，他们都相信“照常推进AI发展会带来不可接受的灾难性风险。”
“但是，”卡罗尔补充道，“我们也不应该把灾难性风险凭空想象成现实——通过有约束力的安全要求、国际协调以及除非有足够的安全进展让我们集体有信心才构建[人工超级智能]的共识，这些风险可以大大降低。”
这正是AI领袖和广大公众现在所呼吁的那种“国际协调”。
“我们从未处于这样一种境地：我们真正理解自己在AI方面驶向何方，”纽约大学的里斯说。“我们一直有这样一个模糊不清、未定义的终态，我们并不真正理解它，但我们必须打败中国才能到达那里。当我们甚至不理解路径、甚至不理解终点线是什么——甚至不知道是否存在终点线时，赛跑真的很难。”

英文来源：

When OpenAI CEO Sam Altman, Anthropic CEO Dario Amodei, Google DeepMind cofounder Demis Hassabis, and SpaceX head Elon Musk loosely agreed over the weekend to slow down AI development, skeptics spotted an ulterior motive immediately. The AI titans had declared that their aim was to “pace the frontier,” signing on at least partially to a proposal for embedding third-party auditors, regulating domestic labs, and reaching a global slowdown agreement. Their critics, however, argued they simply wanted to stop would-be competitors, kneecap the open-source movement, and avoid real legal safeguards — some dubbed it an outright “cartel.”
Is Big Tech’s AI slowdown a safety pact or a cartel?
Major AI companies want to ‘pace the frontier’ — but their proposed solutions need real teeth.
Is Big Tech’s AI slowdown a safety pact or a cartel?
Major AI companies want to ‘pace the frontier’ — but their proposed solutions need real teeth.
The truth is more complicated, according to sources across the industry. The three-step proposal, laid out in an essay by Amodei, is calling for changes long espoused by AI safety advocates. While it could become a substitute for regulation, under Trump, substantial regulation is unlikely anyway. But experts say that on an issue that’s only likely to grow in importance, AI leaders aren’t the best people to lead the charge.
“The industry as a whole needs new champions,” said Nick Reese, an adjunct professor at New York University and the Department of Homeland Security’s former director of emerging tech policy. “We’ve looked at people like Dario Amodei, Sam Altman, and Elon Musk as these people who are closest to the problem and working on it every day, and they’re the ones who know best. But the truth is there’s never been a realistic vision for what we’re building toward.”
“The industry as a whole needs new champions.”
Concerns about AI advancement have been escalating for months, sparked largely by revelations that swarms of agents were behind rogue hacks going on right under the noses of leading frontier labs Anthropic and OpenAI. Reports from both companies fueled the fire, as did the resignation of Anthropic researcher Jacob Coxon, who posted a public letter explaining his choice. “The people building AI earnestly believe that it could kill us all by the end of the decade,” he wrote, adding that neither OpenAI nor Anthropic is “acting responsibly” and rather “racing straight to self-improving superintelligence and gambling with our lives.”
Dire warnings about AI aren’t anything new inside the industry, but Coxon’s letter seemed to break containment. It’s been viewed more than 170 million times on X alone, while Coxon and his story have made the rounds in newspapers and on TV.
To a significant chunk of AI safety researchers and AI nonprofit workers, however, this is simply drawing more attention to an already widely discussed issue. “A lot of people are thinking of this as Dario’s idea, or it’s coming from the CEOs, but that’s false,” said Daniel Kokotajlo, an ex-OpenAI employee who now leads the AI Futures Project, an AI research nonprofit. “People outside the companies have been calling for this for years … There’s been this growing chorus of voices saying, ‘Please don’t build superintelligence soon. We are not ready. You need to slow down.’”
Kokotajlo said that after years of people “yelling at them to do this” — including more than 1,000 AI lab employees signing a public letter from July, which called for a slowdown in AI development after the OpenAI-Hugging Face incident — the CEOs are now “now bowing to that pressure and also claiming credit for it, [although] not rightfully.”
Overall, a handful of sources told The Verge they believe that the verbal agreement made by the AI leaders is a step in the right direction. NYU’s Reese said he “do[es] not think it’s all hot air.” Apollo Research CEO Marius Hobbhahn called it “a good idea,” saying “it’s one of the best things for safety in a long time if it actually happens.” The Midas Project’s Tyler Johnston said it was a “good sign.” Redwood Research CEO Buck Shlegeris said it was “some great news — obviously it’s hard to know whether this is going to convert into anything real, but I’m feeling cautiously optimistic.” All agreed, though, that there was still a lot of work to do to make this verbal commitment a reality, particularly when it comes to making it an ironclad agreement.
“It’s one of the best things for safety in a long time if it actually happens.”
Many still question the motivations of AI’s leaders. The larger tech industry has spent years getting ahead of regulation by lobbying for its own preferred rules or promising self-regulation. Big platforms have proposed policies that could hit smaller competitors harder, using altruistic language to justify self-serving goals. They’ve been accused of safety-washing, or making meaningless changes that give the false impression of actual safeguards. It’s no surprise people are concerned this will happen in the AI industry as well, particularly since AI labs’ voluntary safety frameworks have been criticized for years.
Several sources believe that concerns of safety-washing are valid. “There’s a serious concern that they’re not actually going to slow down,” Kokotajlo says, adding that the fear is that “they’ll just bring in some external auditors, do a bunch of safety paperwork — some of which will be genuinely good — but at the end of the day, it actually won’t slow them down very much at all.”
NYU’s Reese compared this gambit to the social media platforms’ playbook a decade ago, when companies began calling for regulatory action to get ahead of impending, less favorable laws. For the AI industry, Reese said, “the hammer may not come in this administration, but I think if there were a Democratic administration after the next election, there would be a really good possibility.”
That kind of regulation would be vital, said Sacha Haworth, executive director of the Tech Oversight Project. She said any voluntary framework is essentially regulatory capture and that “we should not be letting the foxes run the henhouse. This is not an opportunity for Congress to once again outsource responsibility to industry.” Daniel Lobo-Lewis, co-founder of the Political Integrity Project, said that voluntary regulation will likely go the way of Meta’s largely toothless Oversight Board.
Most people also, however, believe companies are in no imminent danger of regulation, besides the model pre-release review periods AI labs have agreed to under the Trump administration. President Trump posted on Monday that “the only control or ‘guardrails’ that AI needs is a STRONG AND SMART (High IQ!) PRESIDENT, and the U.S.A. has that, in spades!” He also called Nvidia CEO Jensen Huang while Huang was onstage at a conference — Trump told the crowd on speakerphone that recent concerns about AI were a “hoax” and that “the robots will not be taking over.”
As for concerns that regulation could hold back other companies or open source developers, calls for slowdowns have been almost exclusively aimed at big frontier AI labs defined by exact size metrics, many industry experts told The Verge.
With no government AI regulation incoming, the best option may be to push companies into an immediate, measurable, and enforceable agreement. Kokotajlo’s AI Futures Project, for instance, proposed AI labs giving auditors access to their compute budget — and pledging to decrease their compute budget for research significantly, which would then slow down AI advancement and allow other AI labs to catch up. (Amodei’s essay already called for AI labs to allow external third-party auditors — like METR, Apollo, and Redwood Research — to embed within their organizations to some extent and potentially be able to flag, and blow the whistle on, problematic findings.)
“If I am going to die at the hands of killer AI, I want it to be American, not Chinese.”
Arguably the single biggest challenge to an AI slowdown can be summarized in one word: China.
AI leaders and politicians alike have long positioned China as the reason why US AI progress can’t slow down — because no matter how dangerous the technology may become, they’d rather it be in US hands than Chinese ones, and China won’t pump the brakes. Lobo-Lewis compared China fears to the Cold War missile gap. One X user wrote, “If I am going to die at the hands of killer AI, I want it to be American, not Chinese.”
Redwood Research’s Shlegeris said it wasn’t in the best interest of neither the US nor the Chinese government to pursue AI development in a “reckless” way, adding that “it would not be unprecedented levels of international coordination.”
“[People are] taking for granted that China won’t cooperate,” Johnston said, adding, “This is an issue so serious and so widespread that it seems like it’s in everyone’s interest to coordinate on it, in the same way it was in both the US and Russia’s interest to coordinate on nuclear de-proliferation.”
On Monday, Chinese Foreign Ministry spokesperson Guo Jiakun did push back on the calls for a slowdown, calling it “fearmongering.”
The Midas Project’s Johnston and Redwood Research’s Shlegeris both said that even in the absence of Chinese cooperation, it’s still vital to encourage US coordination. And for the Tech Oversight Project’s Haworth, the slowdown presents an opportunity to “position ourselves as the compass for how AI technology can be developed and used.” She sees arguments to the contrary as part of a familiar playbook. “China gets brought up as a bogeyman every time that an industry wants to escape oversight.”
Kokotajlo simply compared the situation to a cartoon he’d seen, with people in a car driving off a cliff. He recalled the speech bubble saying something like, “Hooray, we’re ahead of China.”
“We really do earnestly believe AI could kill all humans!”
The issue underlying all the recent panic is a milestone known as recursive self-improvement, and based on the industry’s current trajectory, we’re only hearing the beginning of the alarm bells.
RSI refers to a potential industry milestone at which point AI models can train, advance, and create new versions of themselves — all without human involvement. Anthropic has said this point could come as soon as early 2027, and OpenAI’s chief scientist wrote earlier this month that OpenAI is directing a lot of resources towards reaching this goal. Increasingly, engineers and researchers in the AI industry worry that RSI precedes a whole host of new and more serious AI concerns, including more severe cybersecurity incidents that more deeply impact society at large.
Amodei cited RSI as a major factor in his decision to call for a slowdown: “since roughly this summer, AI has been advancing drastically faster.” He wrote in his recent essay that RSI is “starting to happen across the industry, including at Anthropic … Left unchecked, it could outrun our ability to understand and control these systems, and so must be pursued very carefully, if at all.” Amodei proposed implementing “some kind of ‘speed limit’” on the rate of RSI and compared it to caps on missile numbers.
In Jacob Coxon’s resignation post from Anthropic, he warned of “superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources.” Many other researchers at leading AI labs echoed his concerns. It’s “hard to overstate how dangerous” it is to speed towards RSI, wrote Jasmine Wang, an OpenAI researcher. Vishal Maini, an ex-Google DeepMind employee, said RSI is “now so imminent that no other option makes sense.”
Fears about RSI can easily turn apocalyptic. “AI developers believe their technology could cause human extinction (or similarly bad outcomes),” Samuel Marks, an Anthropic researcher, wrote on X “This could happen in the next few years. In general, the more senior the employee, the more concerned they are.” Another Anthropic researcher and team lead, Evan Hubinger, wrote on X that “Jacob is correct here—we really do earnestly believe AI could kill all humans!” (He put the chance at greater than 10 percent over the next decade.) Alex Turner, a former Google DeepMind employee, wrote that “many researchers believe they are building something that could kill everyone on the planet. It was literally my day job to think about how to stop that.”
Micah Carroll, an OpenAI researcher, wrote that Coxon’s belief is a “cross-partisan position” with research teams across “all frontier AI companies,” and that they all believe that “business-as-usual AI development poses unacceptable catastrophic risk.”
“But,” Carroll added, “we should also not hyperstition catastrophic risks into existence – they can be greatly reduced via safety requirements with teeth, international coordination, and a consensus to not build [artificial superintelligence] unless there are sufficient safety advances to make us collectively confident to do so.”
That kind of “international coordination” is exactly what AI leaders, and the public at large, are now calling for.
“We have never been in a situation where we actually understood what we were driving towards with regard to AI,” NYU’s Reese said. “We’ve always had this amorphous undefined end state that we don’t really understand but we have to beat China to get to. It’s really hard to race when we don’t even understand the path or even understand what the finish line is — or even if there is one.”