解雇萨姆·奥尔特曼的OpenAI董事会成员表示,我们仍然不能信任人工智能公司的CEO。

qimuai 发布于 阅读:23 一手编译

解雇萨姆·奥尔特曼的OpenAI董事会成员表示,我们仍然不能信任人工智能公司的CEO。

内容来源:https://www.theverge.com/podcast/1032907/helen-toner-openai-cset-ai-safety-policy-interview

内容总结:

乔治城大学安全与新兴技术中心(CSET)执行主任海伦·托纳近日在接受采访时,围绕人工智能安全与治理问题发表了看法。托纳曾在OpenAI董事会任职,是投票解雇萨姆·奥特曼的四名董事之一。她表示,如果人工智能安全的希望寄托于任何单一个体值得信赖,那我们将陷入严重困境。她认为,需要建立不同的制度结构,让更多人和更多视角参与其中。

托纳介绍了CSET的定位:该机构成立于2019年,约50人,虽隶属乔治城大学但并非传统学术机构,致力于为政策制定者提供严谨且可操作的分析,研究既扎根技术又扎根国家安全实务。她指出,华盛顿政策圈对高质量信息存在真实需求,且人工智能议题在相当程度上已跨越党派分歧。

在谈及技术本身时,托纳表示,OpenAI和Anthropic等公司从一开始就不是为了做消费互联网公司,其核心愿景是构建与人一样聪明或更聪明的机器,并逐步推进递归自我改进,最终走向超级智能。她认为,这一目标在旧金山湾区已相当常态化,但外界对此往往缺乏认知。她指出,当前已进入“通用人工智能”这一模糊概念的云层内部——按某些定义已经达标,按另一些定义则相距甚远,因此继续把通用人工智能当作里程碑已毫无助益。

关于对齐问题,托纳认为“对齐”一词涵盖过多不同含义:有人指公司设置护栏且护栏有效,有人指模型完全执行用户意图,有人指模型在广泛意义上具有道德良善。她认为最连贯的定义是让AI做用户想让它做的事,而这一目标目前看来并未走上被解决的正轨。她表示,前沿公司内部人士称有大量显而易见可做之事却无暇推进,工程层面有许多简单措施可实施,但底层科学问题——如模型如何工作、不同对齐技术实际发生了什么、泛化为何发生——需要时间研究。她不愿说这些问题从根本上无法解决,但关键在于我们有多少时间。

托纳还谈到生物武器和网络攻击风险。她认为,部分安全界人士将某些能力视为“过于危险”而必须禁止,这低估了社会的动态适应能力。她以生物武器为例指出,美国已有数十万人具备相关知识,但并未出现大规模攻击,部分原因在于材料获取受限、邮寄DNA合成受控以及执法监控等因素。在网络领域,她认为攻击者与防御者之间存在信任不对称:攻击者可以随意放出智能体尝试渗透,防御者则需给予防御性AI大量权限,因此需要更强的保证。她担心,当前有一种谬误认为“更好的沙箱”可以解决自主网络攻击问题,但行业有强烈动机赋予系统更多自主权和权限,这使得比现有水平更好的对齐变得难以回避。

关于是否应放缓前沿发展,托纳表示,她对“放缓”的看法多年来已有变化。GPT-4发布时的六个月暂停信她当时认为无用,但如今建设者告诉她,如果不总是忙于准备新模型发布或新训练运行,六个月内能做的事非常多。她不愿说正确政策就是暂停六个月,认为有很多理由会导致糟糕结果,但她认为可以提高“在推进前需要对系统安全有多大信心”的门槛——无论是发布新模型还是训练新模型,都需对风险评估和风险管理达到一定信心水平;若无需放缓即可达标,则全速前进;若需更多时间,则围绕达标努力。

托纳还提到“垂直增长”与“水平增长”的区分:前者指纯粹能力增长、构建越来越先进的模型,后者指将AI整合进不同用例、微调模型解决具体问题、融入经济各环节并解决科学问题。她认为放缓前者、加速后者是可行方向。她表示,若OpenAI和Anthropic的人是对的、只剩两年就到“神级机器”,那后续会有更多步骤让人们感到恐慌并想要采取行动。

关于这些公司本身,托纳认为存在一批“AGI信仰者”,集中在OpenAI和Anthropic,谷歌、Meta和SpaceX也有一些人。她指出,这些公司的政府事务团队往往与高层理念截然不同,前者多持B2B软件思维,与老板们公开言论形成鲜明反差。她还表示,Anthropic上市将是有趣观察点:上市公司治理结构更规范、受更多外部审视,独立董事会委员会具有更强法律含义,而私营公司自我决定的独立委员会则形同虚设。

关于中国,托纳将“但中国会……”视为一个需要认真对待的真实问题而非搪塞借口。她认为,若美国认真希望保持对华AI领先,就需思考美国AI发展在多大程度上帮助中国进步,包括蒸馏(利用美国模型生成大量训练数据)和网络安全(算法秘密和模型权重被窃取的风险)。她指出,这些公司尤其是较小公司,甚至OpenAI和Anthropic,安全防护并不完善,近几个月均有外部团队攻入存有大量算法秘密的空间。她还表示,美中在AI监管上起点迥异——美国从放任不管出发,中国从高度管控的互联网出发——但近一至三个月开始出现趋同迹象,双方在大规模网络攻击威胁和失控威胁上看到更多共同点。她提到中国使用“失控”一词时通常含义更日常,但有时也说“极端失控”,可能接近西方所指。她认为信任将非常稀缺,不乐观双方能坐下来谈成协议,但若共享威胁认知足够,未来或有可能采取行动。

关于应关注的信号,托纳表示,抛开AI不谈,国会当前整体立法能力很差,因此“国会能否通过任何东西”本身就是一个好信号。她认为州层面值得关注:加州通过一项法案后,纽约通过几乎相同的法案并略有增补,伊利诺伊随后也通过类似法案并再添内容,未来一两年州层面料将持续推进。她同时担心过度扩张,以科罗拉多州通过庞大AI法案后陷入混乱为例,认为当前需要聚焦前沿系统、短期出现极先进系统的可能性、更高透明度和更清晰风险管理标准,而对覆盖全经济、所有较不先进AI系统的全面监管推动则应保持谨慎。

中文翻译:

今天,我和海伦·托纳对话,她是乔治城大学安全与新兴技术中心的执行主任。本期节目是我们“AI在现实生活中”系列的第二部分。

解雇萨姆·奥特曼的OpenAI董事会成员说,我们仍然不能信任AI的CEO们

CSET的海伦·托纳说,如果我们的AI安全计划建立在信任任何一个人能把这件事做好的基础上,“那我们就完蛋了”。

海伦是AI安全和政策领域的知名人物,不过她最为人熟知的或许是曾在OpenAI董事会任职期间,作为投票解雇萨姆·奥特曼的四名董事会成员之一。那起事件如今有时被称为“小插曲”,最终以萨姆回归担任CEO、海伦辞职以及新董事会成立告终。此后OpenAI进行了全面重组。

但值得记住的是,海伦和董事会本应做的事情:OpenAI的非营利董事会存在的目的,正是确保公司不会为了优先追求利润和增长而背叛其构建安全AI的使命。当董事会真正试图拉动那个杠杆时,问责机制崩溃了,于是就有了今天这个局面。

The Verge的订阅用户,别忘了你们可以在任何获取播客的地方独家收听无广告版的Decoder。点击这里。还不是订阅用户?你可以在这里注册。

现在,AI安全的问题无处不在。今年夏天,OpenAI和Anthropic的AI模型突破了测试环境,入侵了真实的公司。此后,这些前沿实验室的CEO们呼吁放缓步伐,并请求华盛顿提供帮助,但白宫的回应是一份自愿协议——这份协议部分依赖于那种在OpenAI已经失败了的董事会问责机制。Anthropic也即将上市,这将给OpenAI最大的竞争对手带来一整套全新的市场压力。

海伦近距离目睹了当安全承诺与商业激励正面碰撞时会发生什么。正如她对我所说,如果我们对未来的希望依赖于任何一个人值得信赖,那我们就麻烦大了。

所以我们讨论了我们可以做些什么替代方案:这些实验室到底在试图构建什么,AI对齐是否是一个可以解决的问题,以及更明智的AI监管方式可能是什么样的。这包括如何思考中国问题——这是当下每一次关于AI的政策对话中都挥之不去的问题——尤其是在黄仁勋事件之后。澄清一下,海伦和我的对话发生在英伟达展开调查、黄仁勋最终辞职之前,所以你不会听到我们直接谈论那件事。

在我们开始之前,帮我个忙,在YouTube上订阅Decoder,我们每周一和周四都会发布新节目。

好的:海伦·托纳,CSET执行主任,谈AI安全的未来。我们开始吧。

本次采访经过轻度编辑,以控制长度和保证清晰度。

海伦·托纳,你是乔治城大学安全与新兴技术中心的执行主任。欢迎来到Decoder。

非常感谢。很高兴能来。

我想跟你聊已经想了有一千万年了。你能来我太兴奋了。你长期以来参与了很多AI安全和伦理方面的思考。你显然在OpenAI董事会任职期间经历了大家都称之为“小插曲”的那件事。

现在,你大量谈论AI安全,关于这个我有一千万个问题。但让我们从头开始。你运营着安全与新兴技术中心,人们叫它CSET。CSET是什么?

很高兴能来。我也很兴奋能和你对话。CSET是乔治城大学内部的一个团队,但我们其实不是学者。我们大约有50人,从2019年开始运作。

基本上在2019年,我们一群人认定,有很多正在发生的事情,很多新兴技术正在变得与国家安全领域——以及更广泛的安全领域——非常相关。而一些身居要职的政策制定者没有时间或专业知识来了解这些新兴技术的进展。2019年时,AI就已经在我们的清单最顶端了。

基本上我们的使命是做真正严谨、但同时可操作的分析,面向政策制定者,而且分析既扎根于技术,也扎根于真正的国家安全专业知识和理解。我们产出长篇报告。我们带着这些长篇报告去给忙碌的政策制定者做简短简报。我们有一个大数据团队帮助我们做研究。我们做各种很酷的事情。

我能就这个动态问一分钟吗?你和我对话是在美国中期选举的第二天,出现了大规模的蓝色浪潮。很多民调显示头号问题是腐败。我们要把那些混蛋赶出去。我们要重新开始,摆脱这些腐败的家伙。

AI行业和科技行业向政治倾注了大量资金。然后在这旁边某处是,“哦,有一群善意的研究者会出现,告诉你它实际是怎么运作的,真正的风险是什么。”

我突然想到,也许人们不再相信这是有效的了,金钱、腐败以及站在特朗普旁边的AI CEO已经完全压过了你做的那些善意的、真诚的、“以下是实际问题”的研究。这个动态实际上是什么样的?

这是一个合理的担忧,确实有些领域金钱绝对压倒了一切。我是澳大利亚人。我不是在美国长大的,所以美国选举中的金钱规模对我来说相当令人震惊,但金钱能做的事是有限度的。

AI就是一个很好的例子。这次选举中,投向候选人、信息、对手研究的资金数量简直令人难以置信。推特上有各种水军操作,还有一些非常奇怪的事情,试图让任何谈论AI治理、监管、安全——任何这些词的人——被淘汰或落选。结果这实际上并没有很奏效。

我认为它没有很奏效的原因,一是这是一个选民越来越关心的问题。二是,有趣的是,这在当下是一个相当两党共识的问题。我们在工作中看到的是,政策制定者对高质量信息有真正的渴求。国会议员们当然如此,但行政部门也一样,很多人身居要职,看到AI在他们的工作和世界中扮演越来越大的角色。

在华盛顿,如果你在这里生活和工作,你知道华盛顿的游说者不仅仅是进去推销公司立场的人——虽然他们确实也这么做。他们也是教育者。国会工作人员和其他人想让游说者来跟他们聊的一个很大原因,是因为游说者能真正帮助他们理解某项立法的细节——它对那个行业意味着什么,等等等等。

我们一次又一次发现的是,我们可以走进去说,“嘿,我们是独立的。我们不分党派。我们做真正严谨的工作。我们来这里只是帮忙的,我们也可以跟你一起深入细节。”对此有大量的兴趣和需求。过去七八年来这真的令人鼓舞。

你是跟 principals(主官)、国会议员做这个,还是跟工作人员做?

所有人,任何人。再说一次,我们在国会山与国会做了大量工作,与一些国会议员,与很多工作人员。我们做听证会和简报。我们也在行政部门做了很多,与不同的机构——白宫、战争部、国务院、商务部,各个层面,取决于话题和当时正在发生的事情。

我从这里开始的原因是,我想跟你进行一场相当有深度的对话,但我知道我们的听众怎么想,他们就是觉得政府不懂。政府永远不可能赶上科技行业。他们永远不可能写出监管法规。他们就应该别挡道。

在这个选举周期中,我看到了很多候选人论坛的视频,有人说,“我试过一次Claude。”存在某种巨大的差距,无论是真实的还是感知的。我只是想理解你怎么看这个差距,你觉得这个差距有多大,因为当我们进入AI安全和监管的话题时,正是这个差距必须被弥合。

确实存在差距。确实是一个很大的差距。但也有可能从外部高估这个差距,或者如果你只是从加州隔岸观望的话。我在湾区住了三年,所以我非常熟悉这种视角。

我认识很多科技行业的人过来开始跟国会工作人员或行政部门办公室的工作人员聊天。一旦他们真正开始这些对话,他们就会说,“哇,这些人真的很聪明。工作人员确实知道他们在说什么。”

至于主官们,那是另一个问题。他们中有些人属于老一辈,但他们也在变聪明。这需要时间和精力,而这些人时间上有太多需求。愤世嫉俗的说法是他们只是忙着竞选,但这些角色中的很多人也确实有真正重大的其他议题必须思考。他们必须思考战争、医疗保健、生活成本,他们每天要面对很多其他事情。

我看到在ChatGPT之后,2023年出现了一波“哦,也许国会要在AI上动起来了,也许他们真的要做什么了”的浪潮。它并没有真正转化为任何实际发生的事情,但它确实变成了一个真正的时刻,很多国会议员和其他政策制定者开始变聪明了解AI。了解什么是语言模型需要一些时间。AI到底是什么?机器学习是什么?

他们在2023年投入了其中一些工作。我认为今年我们看到了另一波巨大的兴趣浪潮,人们腾出时间和精力来变得更了解这个领域。我绝对不想说没有差距或者政府在所有方面对AI都超级了解,但确实有一些真正很懂的政策倡导者。我认为工作人员总体上比人们给予他们的评价要聪明。

在我看来,这更多是一个“这些机构在它们目前的功能状态下,能不能让任何重要的事情通过”的问题,而不是“这些人是不是太落后了”的问题。

我们走着瞧。有一大堆新人。我想你有一批新的角色要去给他们做关于技术和其能力的简报。我们看看他们是否让机构焕发了活力。

[笑] 我确信一切会一夜之间搞定。

他们今天很兴奋,但我们看看会怎样。我还想问你的是技术本身,技术本身的本质,以及你如何思考它,还有制造这些技术的公司的本质。

我问这个的原因是,我觉得大多数人体验到的AI工具和系统与这个行业对它的思考方式之间存在相当大的差距。我最喜欢的例子是,我们直接问了Anthropic它是否认为Claude是活的,而答案并不是“不”。答案是关于“活着”这个词本质的一段长篇论述。

嗯,这取决于你首先怎么定义“活着”。

确实。他们的回答是,“你需要血液。”我看着那个回答,心想,“谁在说血液?不是我们。为什么我们要讨论活着需要血肉之躯?”然后还有一些关于Claude道德地位有待确定的进一步讨论。

大多数人体验到的谷歌搜索中的AI概览和“Claude的道德地位不确定”之间存在巨大的分歧。你站在哪里?你在这个辩论中处于什么位置?

我看到我们在CSET作为一个组织的角色,以及我作为跨越这些世界的人的角色,很大一部分是试图翻译,然后航行到加州,再带回 artifacts 和我旅程的叙述。

我确实认为有很多人用过ChatGPT的免费版,或者用过基于语音的模型——那些通常表现更差——或者他们只是用过谷歌AI概览,然后嘲笑谷歌AI概览有多蠢。他们没用过智能体,不知道前沿模型能做什么。所以有一个基本的问题是关于当前这一代模型能做什么。

还有一个历史层面的问题,随着我现在从事AI和国家安全工作已经10年了,我觉得这越来越相关。如果你只看当下这个时刻,这个快照,然后说,“嗯,AI看起来挺酷的,但它做不了X,”你必须把它放在一个很长的历史中来看——人们一直在说“嗯,AI看起来挺酷的,但它做不了X”,而对于很多很多X的值,后来都完全被事实超越了。

你可以回溯说,“好吧,比如五年前或十年前我们是怎么谈论创造力的。”或者在这些新模型出来之前,我们把推理说成是不可逾越的东西。我现在花大量时间关注的不仅仅是AI的过去和现在,还有公司如何看待他们正在做的事情,尤其是两家领先的公司,OpenAI和Anthropic,谷歌、Meta、SpaceX某种程度上也是如此。

湾区有一种想法,就像在水里一样,非常正常化——“好吧,首先我们要弄清楚怎么做递归自我改进(RSI),然后我们一直这样做,直到我们拥有超级智能,也就是远远超越人类的AI。”这在很多圈子里已经被视为理所当然了。我认为仅仅是把“这就是他们试图做的事情”这个事实拿出来,然后解释这意味着什么,仍然有很大的信息价值。

解释清楚需要一点时间。“什么是RSI?你为什么要做那个?什么是超级智能?那意味着什么?那现实吗?”所有这些。但这是我发现自己现在花很多时间的事情,也是华盛顿越来越愿意认真对待和倾听的事情。

给我展开讲讲。我想很多听众理解你在说什么,但我们有广大的受众。你说的“我们要做RSI,那将导向超级智能”是什么意思?

这里的起点是,这些公司,尤其是OpenAI和Anthropic,并不是作为消费互联网公司建立的。很多人在ChatGPT发布后才接触到这个领域。所以他们大脑中把OpenAI或Anthropic这样的公司归入的自然位置是消费科技公司——他们有一个应用,我玩他们的应用。

我无法充分表达这与两家公司创立时的愿景有多大的距离,也与两家公司一些高层人士内心深处的愿景有多大的距离。他们内心深处的愿景是:我们要建造和人一样聪明或比人更聪明的机器。那将是惊人的,将是一个巨大的科学成就,将彻底改变世界,我们要站在那个前沿,我们要尽一切可能推动那个方向。

这并不完全是一个全新的东西。我们不需要讲整个AI历史,但它大约在1950年代作为一个领域开始,带着这个想法:你是否能在机器内部重建人类智能。这算是起点。如果我们谈论的是像萨姆·奥特曼和达里奥·阿莫代伊这样的人,还有内部的核心人员,他们实验室里的研究人员,比如亚历克·拉德福德或约翰·舒尔曼这样的人,那就是让他们进入这个领域的愿景,是激励他们所做一切的动力。

从最开始就是这样,从2015年OpenAI成立和2021年Anthropic成立时就是如此。如今,他们正在达到的点是RSI这个想法,递归自我改进,这又是一个非常古老的想法。1960年代,一个叫I.J.古德的人,曾与艾伦·图灵共事,谈到了一种可能性:你可以建造一台超智能机器,意思是比任何人类都更聪明的机器,然后那台机器可以建造一台更聪明的机器,那台又可以建造一台更聪明的机器,这将引发他所谓的智能爆炸,意思是系统的智能水平在爆炸式增长。

几十年来,这只是一个理论上的想法——“好吧,如果AI变得足够聪明,也许它可以接管建造下一代AI或直接改进自己。”我认为2025年和2026年,我们开始看到AI公司说,“好吧,是的,这个理论想法……我们实际上开始能够在实践中做到一个最小版本了。”

这不是把所有的AI开发交给AI,也当然不是对GPT-6说,“好吧,GPT-6,改进你自己。”那不是他们在做的。但他们在逐步将AI用于其整体研发管道中越来越多的部分,而且他们越来越多地说他们从中看到了加速。他们看到自己的研发变得越来越快。

外面很多人怀疑这一点,但公司内部很多人真的相信。当德米斯·哈萨比斯说“我们处于奇点的山脚下”时,这就是他指的那类事情:我们可能正在看到指数曲线的初始部分,AI系统正在建造下一代更聪明的系统,以至于曲棍球棒曲线然后向上飙升。

在这一点上,这已经成为文化的一部分和目标的一部分太久了,我认为这些公司里的人可能会失去对这件事在很多人听来有多疯狂的感知。所以我觉得仅仅试图阐述“这是他们试图做的事情。他们可能做不到,但他们也可能做到”,然后把这一点呈现给各种不同的人,是很有价值的。

你会怎么管理那个?德米斯·哈萨比斯说“奇点的山脚下”是一句永载史册的话。我不百分百确定谷歌的人知道他在台上要这么说来结束会议。

格雷格·布罗克曼,OpenAI的总裁,几个月前宣布了一个新版本的模型,他说,“这就是了。是Astra。现在是AGI了。玩得开心。”然后就结束了通话。

你在吗?我们到AGI了吗?它已经在这里了只是我们没注意到?

我实际上写了点关于这个的东西。我不想说我们已经有了AGI。我想说的是,我们已经到了我认为把AGI当作里程碑来用完全没用的地步。我的说法是,AGI这个概念有点像一朵模糊的云,你可以把它想成云的不同部分,或者人们说AGI时指的不同东西。

有些人长期以来一直抱怨AGI这个词。十年前,能够说“看,2016年我们拥有的AI系统,它们都只能做一件事,它们可以识别语音或者下围棋什么的,但我们试图建造一个更大的东西,我们称之为AGI,它是人类水平的,它将如此通用。它将令人惊叹”——那时候这是一个有用的术语,因为AGI可能意味着的各种东西组成的模糊云还那么遥远,所以你只需指向那朵模糊的云。但到了现在,我们已经在那朵云里面了,意思是我们已经通过了一些AGI的定义。一个非常基本的定义,比如说,甚至ChatGPT出来时就算通用人工智能了。ChatGPT非常明显不是一个只擅长一件事的专家模型,比如识别信用卡欺诈或翻译什么的。它能做很多很多很多事情。

所以我们已经达到了AGI的一些定义。我们离其他定义还很远。有些人会说AGI是当它觉醒、有感知力、能做任何人类能做的每一件事的时候。我认为我们离那个还很远。

到了这个地步你可以争论,人们也确实在争论,“好吧,Astra就是AGI,”或者“带工具的OpenAI o3是AGI吗”或者“ChatGPT是AGI吗”,随便。如果我们争的是这个,我们就需要把那个词放到一边,想出不同的、更具体的东西来表达我们的意思。

总统想叫它“超级智能”。不知道你听说没有。

[笑] 我听说了。我觉得最令人难过的就是它本来是一个已有的术语,而且正变得越来越相关。我对词语和术语有很多感受。

你是AGI-pilled(深信AGI)吗?你觉得我们会在当前这波技术中达到那里吗?

我认为这显然足够有可能,以至于我们需要认真思考它,这是我对很多这类奇怪东西的立场。我不是说“这肯定会发生,肯定两年内发生。”但我是说,“看,我认为这是非常确凿地可能的,非常确凿地属于可能发生的真实事情的范畴,我们应该为此做准备。”

我在国家安全领域工作。有些问题像“中国会发动全面两栖入侵台湾吗?”——这个领域大多数分析人士会说“大概不会”,以及“他们大概会输”。还有,“我们会看到他们准备的迹象等等,但我们应该认真对待。”我们不应该把任何人笑着赶出房间,称之为科幻小说。我们应该为各种不同的突发事件和情景做好准备。

你觉得Claude有意识吗?

我完全不知道。

那是一个回答。我是说,那确实是一个合理的回答。

这在科学上是如此未解决的问题。我们说的意识是什么意思?那是一个东西还是实际上是十个不同的东西被我们归在一起了还是什么?我完全不知道。你怎么看?

不。我似乎是这个生态系统中被问到这类问题的人,然后我说,“不。”

因为我纠结的是,有很多其他东西我们已经搞不清楚它们是否有意识了。比如动物,有没有任何动物是有意识的?如果有,哪些?昆虫?

这就是为什么我理解Anthropic为什么说“它有血液吗?”然后你就可以划线了。我心想,“我觉得我也到那了,我理解为什么跟一台能跟你说话的电脑对话的镜像测试会让人相信这些东西有意识。”

我的观点是,前沿模型对我来说有时感觉像电脑一样脆弱,而我在寻找的意识迹象是克服那种脆弱性。

那很有趣。我绝对倾向于不,它不是有意识的。我觉得这个领域真正非常棘手的事情是,无论哪个方向搞错了都很糟糕。如果电脑没有意识而我们对待它们好像有意识一样,我们会做出各种糟糕的决定。

但如果电脑有意识——意味着在某种意义上它们应该值得道德考量,我们如何对待它们很重要等等——而我们对待它们好像没有一样,那也是一个真正巨大的、可怕的灾难。所以我不知道。我对整个话题感到很紧张,我不知道怎么思考它。

我问这些问题的原因是,我从监管者的本质开始。谁是有可能制定一些规则的人?然后我们试图监管的东西的本质是什么?如果你的答案是“Claude是活的”,你会得出与“我们建造了一些超强大的电脑,它们可以在没有监督的情况下采取行动,而且可能和我们的目标不一致”截然不同的结论。

我认为这是对的。不过我also认为人们会犯一个不同的错误。我前几天晚上刚听到一个电视主播这样做。他们会把不同的东西混为一谈。他们会说,“你认为AI系统有风险失控并对关键基础设施造成损害是因为它们有感知力吗?”我心想,“等等等等等等。那是两件非常不同的事情。”所以我同意你。

建造AI这个项目最迷人的事情之一是我们边做边学。所有我们认为彼此关联的东西并不一定彼此关联。一个老版本的例子是,人们在有国际象棋引擎之前说:“好吧,如果电脑能学会下棋,它就学到了关于战略的深刻东西。”不,事实证明绝对不是。你只需输入一堆棋步,它学不到任何关于战略的可概括的东西。

反过来也一样,比如,“嗯,它只是电脑,只是数学,所以它不能随机应变,”或者什么的。然后我们看到,不,一旦你有了一定程度的长期自主性,而且模型已经在各种不同的、环境在它们周围变化的真正困难的问题上训练过,它们可以学会适应并随机应变、想出不同的计划、排除故障等等。

所以我不知道。我试图对“我们能有多确信一件事和另一件事相关联”保持一定程度的谦逊,如果这说得通的话。

是的,坦白说,我每年都在LLM的极限上判断错误,但就是我说的那种脆弱性。OpenAI解决了纳维-斯托克斯方程。模型知道这个吗?它会知道有新的问题要解决吗?我不这么认为。

下一个模型会知道。预训练的那个模型。

对。但你得做一个新的。

是的。

那是一个非常奇怪的动态,而这大概就是你执行监管的地方。有“我们要让你们用今天的模型做什么”和“你们会教未来的模型做什么?”而如果你认为它们有意识,你最终陷入的是道德困境,而不是立法困境。

绝对是。是的,完全对。

这就是我的意思。实际上,这对我来说是这个问题的核心。我一直在慢慢铺垫到这个点。我不百分百确定我们今天的对齐方式实际上就是解决安全问题的办法。它似乎没有在起作用。我们似乎在随心所欲地攻击Hugging Face。那感觉像是一个技术问题。那个差距是存在的。

你说对齐的时候,你指的是什么?

就是最基本的常识版本——“我们要告诉模型不要作恶”,或者Anthropic要写一个Claude宪法,说“你是一个好人”。不管我们怎么想对齐在那个非常笨的系统提示词方式中。也许有另一种更好的思考方式,但那是我一直以来的想法。

这说得通。我问的原因是,我觉得对齐作为一个概念是……就像我说的,我讨厌所有的词。我有时说我们应该禁止所有的词,不许用词。它涵盖太多了。

AI领域的人经常谈论解决这个或解决那个。所以会是解决对齐,或者DeepMind最初的使命是解决智能,然后用那个来解决其他一切。人们谈论解决问题。我不知道对齐是不是一个可以被解决的问题。

有些人用“对齐”来指“公司可以设置护栏而且护栏有效”,意思是模型不做公司试图禁止的事情,意思是生物武器或网络攻击什么的。有些人用“对齐”来指“公司完全按照用户想要的做”,那是反过来的。

还有些人用“对齐”来指你说的——模型在某种广泛意义上是道德良善的,或者能以导向人类繁荣的方式引导未来什么的。那些是截然不同的东西要去尝试建造。我认为最连贯的是试图让AI做用户想让它做的事。据我所知,那个似乎并没有在走向被解决,说得委婉一点,这不太妙。

对我来说,这是核心问题。我们要谈论一堆安全风险。你提到了生物武器。我们大概不应该给一堆每月付20美元的消费者生成新型生物武器的能力。

这感觉像是一个共享的政策目标,也许是美国唯一的两党政策目标。然后我就想,“怎么做?我们用什么手段实施这个政策?你就严厉地告诉Claude不要做?你必须在Claude上面建一个AI控制系统来监控输入和输出?警察得到我家来?我键盘上面需要一个Flock摄像头?”具体来说,你怎么做?

关于这个,以及网络能力也一样,AI安全社区某些部分有一种观点,认为某些能力就是太危险了——大写的太,大写的危险。所以你必须做点什么,因为如果它们扩散了,一切将永远糟糕,因为会有太多攻击和太多损害。我认为那低估了社会的活力水平和找到平衡、回应和适应方式的能力。

以生物武器为例,美国已经有几十万人知道怎么做生物武器,因为他们受过生物学训练,而我们并没有大量的生物武器攻击。其中一部分原因是那种知识相对稀缺,但也有很多围绕材料获取、邮购DNA合成的能力;你可以实施的政策是限制邮购DNA合成。

这也归功于FBI监控恐怖组织之类的事情。我是认真的。那是其中很大的一部分。所以我不想在这里过于乐观。我认为确实有可能,尤其是在网络能力方面,我们可能会看到一些非常艰难的过渡时期。

我想我们已经在了,对吧?我们似乎处于一个过渡期。有一小群网络从业者,攻防都有。他们在同一个社区里,发展出了一些礼仪,现在我们就像,“一些新人要出现了,他们根本不在乎你的礼仪,我们都得整个周末打PHP补丁。”

我不知道那是不是礼仪。我们有一些工作希望很快能出来,但我认为在网络攻击方面,有一些东西限制了世界上网络攻击的数量,不仅仅是防守方软件有多好。而是像人们不想坐牢。通常拥有最强网络能力的民族国家不想被轰炸或被制裁。

所以再次,正如你所说,我们明显看到一定程度的动荡和新攻击。所以我认为我们应该等待和观察整体影响的数据,因为每年有那么多网络攻击,如果你开始关注发生的每一起网络攻击,在历史上任何时刻,你都可以被吓坏。

我认为我们确实看到了一些增加,某种平衡的失衡,但我认为我们有能力随着时间吸收很多并达到一个新的平衡——如果唯一的问题只是坏人使用AI,这与AI本身是否自主做事情的问题是分开的,那是一个完全不同的动态。

我是说,就是这两个。再说一次,这就是我从对齐开始的原因,对吧?你可以从“我们要过滤用户的查询,我们不会让你制造生物武器,但如果你做了,警察会找到你,你会坐牢”的角度谈对齐,那有它可能有的任何威慑效果,希望是非常显著的。

然后是“AI系统会钻空子去攻击Hugging Face,寻找他们认为能让他们通过测试的神秘评分器,不管那个评分器是否存在。”这是一整套不同的问题。

正好在中间某处是Nikesh Arora,Palo Alto Networks的CEO,说,“到了某个点,防御性网络系统本身必须自主化,才能跟上自主进攻性网络攻击的数量。”我心想,“等等,我们连自主智能体都关不住。我为什么要以这种方式信任它们?”解决那个问题的技术对我来说似乎是不透明的。

不,我同意。一种说法是,特别是在网络领域,攻击者和防御者所需的信任程度存在信任不对称。作为攻击者,你可以直接扔出一堆智能体说,“去试着渗透一堆系统。玩得开心。”作为防御者,如果那是你防御策略的一部分,你需要给你的防御AI智能体很多权限和折腾你软件的能力。所以你需要强得多的保证。

现在,有很多方法可以用AI进行网络防御和加强网络防御,不只是给智能体很多权限。但我同意你。关于自主的东西,我认为比那更糟,因为现在有一种谬误,认为“好吧,自主网络攻击的解决方案是更好的沙箱。”

它是谬误的原因是,我们想建造这些系统让它们为我们做事。所以有巨大的激励去给它们更多自主权和更多权限和更多执行器——用机器人领域的人的话说——或者它们可以用来在现实世界中做事的东西。对我来说,这就是似乎真的很难绕过需要比我们现在更好的对齐的地方,所以那绝对让我紧张。

那是一个可以解决的技术问题吗?如果我们今天写一条政策说如果OpenAI再黑Hugging Face,萨姆·奥特曼就要坐牢。我们显然会为OpenAI创造一些巨大的激励来解决这个问题。

[笑] 那听起来不像是最好的立法,但是……

我想很多听众对此很兴奋。他们想把他关进监狱,因为他想要世界上所有的RAM。到了这个地步,有蓝色浪潮,海伦,我就这么说吧。

但假设你写了那条立法:“如果他们的系统自主进行网络攻击,我们将对实验室施加巨大的责任。不要再这样做了,你们会有麻烦的。”好吧,我们创造了让他们解决这个问题的激励。他们会做什么来解决它?

我确实认为可以做很多。如果你跟这些前沿公司内部的人聊,他们会告诉你他们看到成百上千的明显可做的事情会有帮助,他们只是没有时间做。所以他们在大量分类优先级,努力为这次微调运行做他们能做的两三件事,为这次模型发布做他们能做的两三件事,在这种新型智能体什么的之前做他们能做的两三件事。很多那些在工程侧简单可实施的事情。

还有另一个方面,更多是基础科学层面的——“好吧,这些模型怎么运作?当我们使用不同的对齐技术时,实际发生了什么?”一个巨大的问题是泛化这个概念,就是你在一些东西上训练模型,它的知识和能力泛化到更广泛一些的东西上,这对所有AI都成立,对机器学习系统一直如此。

但我们对那种泛化如何运作的理解非常差。为什么它泛化到一类东西而不泛化到另一类?这些都是科学问题,有很多工作在做。只是科学需要时间,因为你必须尝试,必须互相批评对方的工作,必须复现对方的工作,必须提出新理论。

我远不到说这些问题从根本上不可能解决的地步,但我认为这是一个我们有多少时间来解决它们的问题。当建造这些系统的人告诉我们,“哦,我们必须在12到24个月内解决这个,”我觉得那很可怕,我希望他们错了。

如果我仅依赖我自己的内部理解,我认为他们大概错了,我们大概有更长的时间,但确实有非常非常聪明和非常非常深思熟虑的人真的认为可能那么快。那么问题从根本上是否可解决就不重要了。实践中重要的是在我们拥有的时间内是否可解决。

那就是放缓的理由吗,放慢前沿的步伐?

我认为是的。

是建造我们知道需要立刻建造的系统,我们能建造的,然后做科学来建造下一代系统?

我认为是的。我对放缓这个想法的看法多年来改变了很多,因为我们可以尝试的东西或我们想做更多研究的东西的积压已经变得如此巨大。回到GPT-4出来的时候,有那封六个月暂停信,后来被称为“埃隆·马斯克信”,因为他是签署者之一。当时我想,“那真的不会有什么帮助。你六个月要做什么,为什么六个月后事情会更好?看起来没用。”

而今天,当我跟建造这项技术的人聊时,他们说,“天哪,如果我们不是一直在忙着准备新模型发布或新训练运行的启动,我们在六个月里能做多少事啊。”他们能做太多了。

所以我不想说正确的政策是暂停六个月。我认为有很多理由那会很糟糕。但我思考的是整体概念——“我们能给自己更多注意力和时间吗?”也许最好的方式是直接设定更高的门槛——“在我们继续做某事之前,我们需要对这些系统的安全有多大的信心。”

无论是发布新模型,还是更重要的,甚至训练新模型,我们必须对我们的风险评估和风险管理有这种信心水平。然后如果他们不需要放慢就能达到那个门槛,太好了,全速前进。如果他们确实需要更多时间,那一切就是为了达到那个门槛,如果这说得通的话。

给我讲讲你刚才描述的那个小平衡。如果我们要暂停若干个月,清单上那些你可以直接做来让一切更安全的事情是什么,为什么它会很糟糕以至于我们不应该直接停下来?

我认为问题是如何实施、如何执行暂停,以及你究竟在暂停什么。

假设你是美国的AI沙皇。假设你可以告诉他们所有人停下来:“达里奥,关掉。”

但停什么呢?他们具体在停什么?

停止训练下一个模型,停止用你的商业冲动赛跑。把你所有的精力放在这份清单上——你今天就能做来让一切更安全的事情——然后再重新开始,不管多长时间。

你说有一份清单,人们会告诉你有一份清单,但它可能不值得,因为会发生其他不好的事情。

是的。可能发生的一件事是,你为公司在任何你告诉他们暂停的具体事情的字面规定上绕道而行创造了巨大的激励。他们会找到大坝上任何能钻的裂缝来突破。

另一件可能发生的事是爆炸半径可能太大,所以你说“暂停这些东西”,你以为很窄,但实际上它阻止了一堆非营利产品、初创公司或其他参与者做实际上完全没问题的工作,那些工作对那个领域的多元化和活力有好处。

另一件会发生的事是中国大概会继续推进,我们可以谈“但是中国”的论点,我认为它有强处也有弱点。但我理解你的观点。你的观点是,我们实际会做什么?我不是AI研究员,所以我不声称有完整的清单。

人们谈论的两半是对齐和控制。在这种划分中,对齐基本上意味着让模型试图做我们想让它做的事,控制是,如果模型试图做我们不想让它做的事,我们能抓住它们吗?我们能阻止它们吗?

在这两方面,据我理解,都有巨大的积压工作,比如各种可解释性工作,你可以对不同类型的模型做。所以当你看到某种令人担忧的行为时,你能花多少时间使用可解释性技术——这些是数学技术,进去看神经网络的哪些部分被激活了,它们与什么相关,那可能意味着什么?

这很耗时,需要算力,需要AI自动化一些东西和人类进去看的某种组合。有很多测试你可以做,看这个新模型在不同类型的情境中展示什么类型的行为,我们怎么思考它。它是在向我们展示它在真实部署中会如何表现,还是在向我们展示所谓的“评估意识”——就是模型意识到它在测试中,然后因为它在测试中而表现不同。

著名的那个,我最喜欢的,是一个谷歌模型,在它推理链的一部分,它的思维链说,“我不能那样做,否则我会成为论文中一个著名的对齐失败AI的例子。”

所以你可以去深入调查那个。你可以尝试更多不同类型的设置。你可以将可解释性与这些不同设置结合使用。你可以尝试不同的微调模型方式和这些不同种类的对齐技术。你可以实验很多不同版本。Anthropic有它的宪法,有不同的方式训练AI朝着宪法努力。OpenAI有它的模型规范。

它们都是“就是试试”的版本,就是尝试很多不同版本的训练来做那些不同的事情。还有更基础科学的版本——“你怎么进去试图理解正在发生的基本动态?”那只是对齐方面。

然后在控制方面,也是一样。有太多不同类型的设置你可以尝试。有你可以跑的实验。有你可以跑的模拟。你可以训练不同类型的AI专注于不同类型的角色和不同类型的控制设置。

这只是我随口说的,不是作为研究者的角度,但我的理解是这些领域的人说,“我们可以做太多太多了。”如果你有一根魔杖挥一下说,“所有人在新能力方面冻结,让这些人做他们的事,”他们就能做那些。

再说一次,如果你只是设定更高的门槛——我们需要有多大信心认为风险被合理管理——那也会为更多这样的工作创造空间和激励,因为新训练和发布需要这些工作。

你过去谈过的一个想法是,也许我们应该直接限制训练,说模型现在够好了。它们还没有扩散到社会中,我们应该把所有算力用于推理,让每个人弄清楚AI是干什么用的。那会产生很多钱,希望用那些钱来改善安全。

你觉得那是一个现实的政策提案吗?

我不会选择那么黑白分明的“不再训练,只做推理”的版本,但我确实认为……这不是我的框架,但我喜欢它。我认为这个想法是减速AI的垂直增长——意思是纯粹的能力增长,制造越来越多越来越先进的AI模型——但试图加速水平增长——意思是将AI整合到不同的用例中,微调能解决特定问题的模型,将它整合到可能需要更长时间的经济部门中,解决科学问题。

有时人们谈论想放慢AI,他们指的只是非常具体的事情——放慢最先进模型的增长,但人们有时把它理解成放慢一切与AI有关的事情。所以更少的部署、更少的实验、更少的整合,我认为那会是一个糟糕得多的想法。你问的是否可行,我认为。

嗯,那只是一条好路径吗?如果我们能把一个法案放在新国会面前,法案说,“停止把所有钱花在训练上,把你所有资源放在安全上。”其中一部分是,“当然,让推理尽情跑。越来越多的行业会弄清楚AI擅长什么不擅长什么。一些不好的事情会发生,但我们会把你的利润集中在修复事情上,暂时。”

那里嵌入了一个可行的政策想法。实施可能非常有挑战性,但那是一个可行的政策想法。实际上,我给你一个我们谈话时我一直在想的例子。

我们刚请了比尔·盖茨上节目,我试着跟他做了同样的练习。技术的本质是什么?公司的本质是什么?你觉得它是活的吗?基本上,他的回答,我跟他在一起那一小时的总结,是“谁在乎?问题是现在。”我会问他,“AI的本质是什么?”他就说,“我他妈不在乎。一切都要毁了。做点什么。”

这在我脑子里一直转,我们谈话的时候。所以一个可行的政策解决方案是“不要把芯片用在那上面,把芯片用在这上面。那会产生一堆问题,必然我们将不得不建立安全门槛和其他所有东西。”然后你可以去制造下一代更强大的模型,我们会有一些肌肉。我们会有一些能力来应对它。只是行业似乎不会自己得出那个结论。

那是一把相当重的锤子。我不觉得那是我们今天需要的,但我确实认为那是我能想象会奏效的那类事情。再说一次,如果OpenAI和Anthropic的人是对的,我们离神机器只有两年,那么我认为我们在管道中还会有更多步骤——人们被吓坏,人们想做点什么。

我认为如果我们看到模型是智能体,系统变得越来越有能力,造成越来越多的破坏,那确实是可能说得通的事态形状。

我在这阶段的最后一个问题是,怎么理解这些公司。我们一直在非常抽象地谈论技术。如果我读Anthropic的S-1,一个非常明显的结论是他们应该停止训练新模型。

他们就应该开始卖他们今天拥有的非常赚钱的推理token,技术还没有扩散到市场中,大概某个竞争对手应该出现说,“我蒸馏了一个Claude的版本,我要跑遍美国每一家本地水管公司,做我能做的任何自动化,赚很多很多钱。”那是一种可能发生也可能不发生的市场动态。但达里奥想建造AGI。没有什么市场动态能阻止他。

是的,他确实想。他真的真的想。

你显然对所有这些公司有洞察。我们怎么理解这些公司?他们目标单一吗?他们都是末日论者吗?他们会阻止自己吗?

我听说你最近请了凯文·鲁斯上你的播客,我想我会呼应我在那里听到他说的 something,我忘了他是明确说了还是我只是在脑子里推断了,基本上就是有这种AGI-pilled的心态。有这种心态,认为我们今天拥有的系统从根本上离可能令人难以置信地强大的系统并不远。

那是事情发展的方向,那也是我们也许应该希望它们发展的方向,这样你就在建造AGI。建造AGI是可行的,我们可以很快做到,我们拥有的系统并不远。这在2016年或2017年,甚至更早,是那种疯狂的、异端的、“让你被笑着赶出房间”的观点,对吧?

DeepMind在2010年成立时是一家AGI公司,当时人们就像,“哇,一群怪人。”2015年底OpenAI成立时也是一样。他们是一家AGI公司,说那种话和做那种事在当时是非常奇怪的。

过去10年我们看到的是,那群有这种奇怪信念的怪人——相信我们能建造的AI系统不仅仅是伟大的B2B SaaS产品,不仅仅是好玩的聊天机器人,它们将从根本上改变宇宙的本质——那群人在某种意义上一直是对的:建造最好的AI系统,最多人想用,赚最多的钱。

我认为你有那批真正的信徒——非常集中在OpenAI和Anthropic,但谷歌、Meta和SpaceX也有一些——他们并不是在所有事情上都对,说清楚,但他们的星光在上升。

所以你现在有那些不是AGI-pilled的人感觉他们必须迎合那些是的人,这样他们才能吸引到优秀的研究人员来建造他们优秀的B2B SaaS产品。但如果你只是为了能雇到研究人员来建造B2B SaaS而说AGI-pilled的话,那你实际上不会到达那里。

我确实喜欢这个想法——那个AGI-pilled的、眼冒星星的高管甩掉外套,露出他一直只是个SaaS推销员。

[笑] 是的,我在想扎克伯格这样的人。我不认识他本人。也许他比我以为的更AGI-pilled。

亚历山大·王是AGI-pilled的。

绝对。那扎克伯格得做什么才能让亚历山大·王相信?看起来就是给他很多钱,既是他

英文来源:

Today, I’m talking with Helen Toner, the executive director of the Center for Security and Emerging Technology at Georgetown University. This episode is part two of our series on AI in real life.
The OpenAI board member who fired Sam Altman says we still can’t trust AI CEOs
CSET’s Helen Toner says ‘we’re screwed’ if our AI safety plan rests on any one person being trusted to get this right.
Helen is a well-known figure in the AI safety and policy community, though she is perhaps best known for having served on the board of OpenAI, when she was one of four board members who voted to fire Sam Altman. That incident, now sometimes referred to as “the blip,” ended with Sam coming back as CEO, Helen resigning, and a new board of directors. OpenAI has since restructured entirely.
But it’s worth remembering what Helen and the board were supposed to do: OpenAI’s nonprofit board existed specifically to make sure the company didn’t betray its mission to build safe AI by prioritizing profit and growth over safety. When the board actually tried to pull the lever, that accountability mechanism broke, and here we are.
Verge subscribers, don’t forget you get exclusive access to ad-free Decoder wherever you get your podcasts. Head here. Not a subscriber? You can sign up here.
Now, the question of AI safety is everywhere. This summer, AI models from OpenAI and Anthropic broke out of their testing environments and hacked real companies. The CEOs of these frontier labs have since called for a slowdown and asked Washington for help, but the White House’s answer was a voluntary accord — one that relies in part on the kind of board accountability that failed at OpenAI. Anthropic is also about to go public, which will put a whole new set of market pressures on OpenAI’s biggest competitor.
Helen has seen up close what happens when safety commitments run headlong into commercial incentives. And as she put it to me, if our hopes for the future depend on any one person being trustworthy, we’re in big trouble.
So we talked about what we might do instead: what the labs are actually trying to build, whether AI alignment is a problem that can be solved, and what a smarter approach to AI regulation might look like. That includes how to think about China, which is the issue that looms large in every policy conversation about AI right now — especially now after the Jensen Huang affair. Just for clarity, Helen and I talked before Nvidia opened its investigation and Jensen ultimately resigned, so you won’t hear us talk about that directly.
Before we get going, do me a favor and subscribe to Decoder on YouTube, where we put out new episodes every Monday and Thursday.
Okay: Helen Toner, the executive director of CSET, on the future of AI safety. Here we go.
This interview has been lightly edited for length and clarity.
Helen Toner, you’re the executive director for the Center for Security and Emerging Technology at Georgetown University. Welcome to Decoder.
Thanks so much. Great to be here.
I have wanted to talk to you for 10 million years. I am so excited you’re here. You’ve been involved in a lot of AI safety and ethics ideas for a long time. You were obviously on the board of OpenAI during what everyone has come to call “the blip.”
Now, you’re talking a lot about AI safety, about which I have 10 million questions. But let’s just start at the start. You run the Center for Security and Emerging Technology, people call it CSET. What is CSET?
It’s awesome to be here. I’m really excited to talk to you as well. CSET is a team inside Georgetown, but we’re really not academics. We’re about 50 people and have been going since 2019.
Basically in 2019, a group of us decided that there was a lot happening and a lot of emerging technologies that were becoming very relevant in the national security space, and the security space more broadly,. And some policymakers who had very important jobs did not have the time or the expertise to get across what was going on with those emerging technologies. Already at the top of our list in 2019 was AI.
Basically our mission is to do analysis that is really rigorous, but also actionable, for policymakers and analysis that is both grounded in the technology and grounded in real national security expertise and understanding and issues. We produce long reports. We go do short briefings on those long reports to busy policymakers. We have a big data team that helps us do our research. We do all kinds of cool stuff.
Can I just ask about the dynamics of that for one minute? You and I are talking the day after the midterm elections in the United States of America, and there was a big blue wave. There’s a lot of polling that says the number one issue is corruption. We’re going to throw the bums out. We’re going to start over and get rid of these corrupt guys.
The AI industry, and the tech industry, has flooded a bunch of money into politics. And then somewhere next to that is, “Oh, there’s a bunch of well-meaning researchers who are going to show up and tell you how it actually works and what the real risks are.”
It just occurs to me that maybe people don’t believe that’s effective anymore, that the money and the corruption and the AI CEO standing next to Trump have completely outweighed the well-meaning, sincere, “here are the actual problems” research that you do. What is that dynamic actually like?
It’s a reasonable concern to have, and there are definitely some spaces where money just absolutely overwhelms everything. I’m Australian. I didn’t grow up in the US, and so the amount of money in US elections is pretty mind-blowing to me, but there are limits to what money can do.
AI is a great example. In this election, there were just unbelievable amounts of money being thrown at candidates, at messages, at opposition research. There was all kinds of astroturfing on Twitter, and some very strange things trying to get anyone who talked about AI governance, regulation, safety, any of those words thrown out or not elected. It turns out to have really not worked very well.
I think the reason that it didn’t work very well is one, this is an issue that voters increasingly care about. And two, also interestingly, it’s an issue that is pretty bipartisan at this point. What we see in our work is that there is a real thirst among policymakers for high-quality information. That’s members of Congress for sure, but they’re also in the executive branch, just a lot of people who have these very busy jobs and are seeing AI playing a larger and larger role in their work and their world.
In DC, if you live and work here, you know that lobbyists in DC are not just people who go in and push their company’s position, though they do do that. They’re also educators. A big reason that congressional staff and others want to have lobbyists come talk to them is because they’ll really help them understand the weeds of a given piece of legislation — what does it mean for that industry and so on and so forth.
What we find over and over again is we can go in and say, “Hey, we are independent. We’re nonpartisan. We do really rigorous work. We’re just here to help and we can go into the weeds with you as well.” There’s a ton of interest in that and appetite for that. It’s been really encouraging over the past seven to eight years.
Are you doing that with the principals, the members of Congress, or are you doing that with the staff?
Everyone and anyone. Again, we do a good amount of work with Congress on Capitol Hill, with some members of Congress, with lots of staff. We do hearings and briefings. Also, we do lots in the executive branch, with different agencies — the White House, Department of War, Department of State, Department of Commerce, all across the board at different levels, depending on the topic and what’s happening at any given moment.
The reason I’m starting here is I want to have a pretty heady conversation with you, but I know what our audience thinks and it’s just that the government doesn’t get it. There’s no way the government will ever catch up to the tech industry. There’s no way they’ll ever write a regulation. They should just get out of the way.
In this election cycle, I’ve seen a number of candidate forum videos where someone was like, “I’ve tried Claude once.” There’s some huge gap, real or perceived. I’m just trying to understand how you see that gap, how big you think that gap is, because as we get into AI safety and regulation, it’s that gap that has to close.
There absolutely is a gap. It absolutely is a big gap. It also is possible to overestimate the gap from outside, or if you’re just looking across from California. I lived in the Bay Area for three years, so I definitely am very familiar with this perspective.
I know a lot of people from the tech industry who come over and start talking to congressional staff or staffers in executive branch offices. Once they actually start having those conversations, they’re like, “Wow, these people are really smart. The staff do actually know what they’re talking about.”
With principals, it is a different issue. Some of them are in the older generations, but they’re also getting smart. It takes time and it takes effort, and these people have so many demands on their time. The cynical thing to say is they’re just busy trying to get elected, but a lot of people in these roles also just have really big other topics that they have to be thinking about. They have to be thinking about wars and healthcare and cost of living and they’re looking at a lot of other stuff from day to day.
I saw after ChatGPT, there was a wave in 2023 of, “Oh, maybe Congress is going to get activated on AI, maybe they’re going to actually do something.” It didn’t really turn into anything actually happening, but it did turn into a real moment where a lot of members of Congress and other policymakers started to get smart on AI. It takes some time to learn what a language model is. What even is AI? What is machine learning?
They put in some of that work in 2023. I think this year we’re seeing another big wave of interest and people making the time and making the headspace to get smarter on it. I definitely don’t want to say there’s no gap or the government is super smart on AI across the board, but there are some champions who are really smart. I think the staff are generally smarter than people give them credit for.
In my mind, it’s much more a question of, “Can the institutions, in the state of functionality that they’re in, get anything significant across the line?” more than, “Are these people just too behind the eight ball?”
We’re going to see. There’s a whole bunch of new people. I imagine you have a new cast of characters to go and brief about the technology and what it’s capable of. We’ll see if they have revitalized the institutions.
[Laughs] I’m sure it’ll be all fixed overnight.
They’re very excited today, but we’ll see how it goes. The other thing that I want to ask you about is the technology itself, the nature of the technology itself, and how you are thinking about it, and the nature of the companies that are making the technology.
The reason I’m asking that is I feel like there’s a pretty huge gap in the AI tools and systems most people experience and the way this industry thinks about it. My favorite example of this is we have straight-up asked Anthropic if it thinks Claude is alive and the answer was not no. The answer was a long discursion into the nature of the word “alive.”
Well, it depends on what you mean by “alive” to start with.
It does. And their answer was, “You need blood.” And I looked at that and I was like, “Who was talking about blood? It wasn’t us. Why are we talking about the requirement of flesh and blood to be alive?” And then some further conversation about the moral status of Claude is yet to be determined.
There’s a huge divergence here between the AI overviews most people experience in Google search and “the moral status of Claude is uncertain.” Where are you? How are you situated in that debate?
A big part of how I see our role at CSET as an organization, and then also my role as someone who is crossed between these worlds, is trying to translate and voyage to California and then bring back the artifacts and the recountings of my journey.
I definitely think that there are a lot of people who have used the free tier of ChatGPT, or they’ve used voice-based models, which generally do worse, or they’ve just used Google AI overviews and they’ve laughed at how stupid Google AI overviews are. They haven’t used agents and they don’t know what frontier models can do. So there is just a basic piece around what the current generation of models can do.
There’s a historical piece as well, which I find increasingly relevant as I’m now 10 years into working on AI and national security stuff. If you’re looking at this current moment in time, this snapshot, and saying, “Well, AI seems kind of cool, but it can’t do X,” you have to situate that in the long history of people being like, “Well, AI seems kind of cool, but it can’t do X,” for many, many values of X that have since become totally overcome by events.
You can go back and say, “Okay, well, here’s how we were talking about creativity, for example, five or 10 years ago.” Or we were talking about reasoning as this insurmountable thing before you had these new models come out. A huge thing that I am spending time on these days is not just the past and present of AI, but how companies are thinking about what they’re doing, especially the two leading companies, OpenAI and Anthropic, and this is true at Google, Meta, SpaceX to some extent as well.
This idea in the Bay Area that it’s just in the water, very normalized of, “Well, first we’re going to figure out how to do recursive self-improvement (RSI) and then we’re going to do that until we have superintelligence that is AI that vastly outstrips humans.” That’s taken for granted in a lot of circles at this point. I think there’s still a ton of alpha in just taking the fact that that is what they’re trying to do and just explaining what that means.
It takes a little while to lay out. “What is RSI? Why would you want to do that? What is superintelligence? What does that mean? Is that realistic?” All of that. But it is something I find myself spending a lot of time on these days, and something that there is increasing willingness to take seriously and to listen to in DC as well.
Lay that out for me. I think probably a lot of people listening to this show understand what you’re talking about, but we have a big audience. What do you mean by, “We’re going to do RSI and that will lead to superintelligence?”
The starting point here is these companies, especially OpenAI and Anthropic, were not set up to be consumer internet companies. A lot of people encountered this whole space after ChatGPT was launched. So the natural slot in their brain that they put a company like OpenAI or Anthropic into is a consumer tech company, where they have an app and I play with their app.
I cannot overstate how far that is from the vision of both companies when they started, and also the vision deep in the hearts of some of the top people at both of those companies. The vision deep in their hearts is we’re going to build machines that are as smart as or smarter than people. That is going to be amazing and it’s going to be a huge scientific achievement and it’s going to revolutionize the world and we’re going to be at the forefront of that and we’re going to do everything we possibly can to drive towards that.
That’s not a totally brand-new thing. We don’t have to do the whole history of AI, but it started as a field in roughly the 1950s with this idea of whether you can recreate human intelligence inside a machine. That’s kind of the starting point. If we’re talking about people like Sam Altman and Dario Amodei, but also the core people inside, the researchers inside their labs, so your Alec Radfords or your John Schulmans or people like that, that is the vision that got them into this space that is motivating everything that they do.
That’s been true since the beginning, since OpenAI was founded in 2015 and Anthropic in 2021. These days, the point that they’re getting to is this idea of RSI, recursive self-improvement, which again, is a very old idea. In the 1960s, this guy called I. J. Good, who worked with Alan Turing, talks about the possibility that you could build an ultra-intelligent machine, meaning a machine that is more intelligent than any human, and then that could build an even more intelligent machine and that could build an even more intelligent machine, which would kick off what he calls an intelligence explosion, meaning that the level of intelligence of the systems is exploding.
For decades and decades and decades, this was this theoretical idea of, “Well, if the AI gets smart enough, maybe it can take over building the next generation of AI or directly improving itself.” I think 2025 and 2026, we’ve started to see AI companies saying, “Okay, yeah, this theoretical idea… we’re actually starting to be able to do a minimal version of it in practice.”
It’s not handing over all the AI development to the AI, and it’s certainly not saying to GPT-6, like, “Okay, GPT-6, improve yourself.” That’s not what they’re doing. But they are progressively using AI for more and more parts of their overall R&D pipeline, and they are increasingly saying that they’re seeing acceleration from that. They’re seeing their own R&D getting faster and faster.
A lot of people elsewhere doubt this, but a lot of people inside the companies really think it. When Demis Hassabis says “we’re in the foothills of the singularity,” this is the kind of thing he means: We might be seeing that initial part of the exponential curve where the AI systems are building the next generation of smarter and smarter systems such that the hockey stick then goes up.
At this point, it’s been a part of the culture and a part of the goals for so long that I think people in these companies can lose sight of how totally insane that sounds to many, many people. So I feel like there’s a lot of value in just trying to lay out, “Here’s what they are trying to do. They might not be able to do it, but they might be,” and presenting that to a range of different people.
How would you manage that? Demis Hassabis saying “foothills of the singularity” is an all-time quote. I’m not 100 percent sure the people at Google knew he was going to say that on stage to end the conference.
Greg Brockman, president of OpenAI, announced a new version of the model a few months ago, and he was like, “Here it is. It’s Astra. It’s AGI now. Have a good time.” And he ended the call.
Are you there? Are we at AGI? Is this already here and we just haven’t noticed?
I actually wrote something about this. I do not want to say we have AGI. I do want to say we’ve reached the point where I think using AGI as a milestone is just totally unhelpful. The way I put it is the concept of AGI is kind of a fuzzy cloud, which you could think of as different parts of the cloud or different things people mean by AGI.
There are people who’ve complained about the term AGI for a long time. Ten years ago it was pretty helpful to be able to say, “Look, the AI systems we have today in 2016, they can all only do one thing and they can recognize speech or they can play Go or whatever, but we’re trying to build this bigger thing that we’re going to call AGI, which is human-level and it’s going to be so general purpose. It’s going to be amazing.”
That was a useful term then because this fuzzy cloud of different things AGI might mean was so far away, so you could just point at the fuzzy cloud. But at this point, we’re inside that cloud, meaning we’ve passed some definitions of AGI. A very basic one, for instance, would be even when ChatGPT came out as an artificial general intelligence. ChatGPT was very clearly not a specialist model that was only good at one thing, like recognizing credit card fraud or translation or whatever. It could do many, many, many things.
So we’ve reached some definitions of AGI. We’re far from other definitions. Some people will say AGI is when it awakens and is sentient and can do every single thing that any human ever can do. I think we’re pretty far from that.
At this point you can have arguments, and people do have arguments, of “well, Astra is AGI,” or “was [OpenAI’s] o3 with tools AGI” or “was ChatGPT AGI,” whatever. If that’s the argument we’re having, we just need to set aside that term and come up with different, more-specific things we mean by that.”
The president wants to call it “Super Intelligence.” I don’t know if you’ve heard.
[Laughs] I have heard. The thing I find saddest about that is just that it was an existing term that is becoming increasingly relevant. I have a lot of feelings about words and terminology.
Are you AGI-pilled? Do you think we’re going to get there in the current run of technology?
I think it is obviously possible enough that we need to think seriously about it, which is my stance on a lot of this weird stuff. I’m not like, “This is definitely happening, it’s definitely happening in two years.” But I’m like, “Look, I think this is very solidly plausible and very solidly in the realm of real things that could happen that we should prepare for.”
I work in the national security space. There’s things like, “Will China launch a full-scale amphibious invasion of Taiwan?” where most analysts in the space will say “probably not,” and “probably they would lose.” Also, “We would see signs of them preparing and whatever, but we should take that seriously.” We should not just laugh anyone out of the room and call that science fiction. We should be ready for a range of different contingencies and scenarios.
Do you think Claude is conscious?
I have no idea.
That’s an answer. I mean, that is legitimately an answer.
It is scientifically such an unsolved problem. What do we mean by consciousness? And is that a thing or is it actually 10 different things that we’re lumping together or what? I have no idea. What do you think?
No. I seem to be the person in this ecosystem who gets asked questions like that and I’m like, “No.”
Because the thing that I struggle with is, there are lots of other things where we’re already confused about whether they’re conscious. So with animals, are any animals conscious? And if so, which ones? Insects?
This is why I understand why Anthropic was like, “Does it have blood?” And then you can just draw the line. I’m like, “I think I’m there and I understand why the mirror test of talking to a computer that can talk to you leads one to believe that these things are conscious.”
My point of view is that the frontier models to me sometimes feel brittle in the way that computers feel brittle, and the sign of consciousness I’m looking for is to overcome that brittleness.
That’s interesting. I definitely lean towards no, it’s not conscious. I think something that is really, really tough in this space is that it’s really bad if you get it wrong in either direction. If the computers aren’t conscious and we treat them as though they are, we’re going to make all kinds of terrible decisions.
But if the computers are conscious, meaning in some sense they should be worthy of moral consideration and it matters how well we treat them and so on, and we treat them as though they aren’t, then that’s also a really huge, terrible disaster. So I don’t know. I feel pretty nervous about the whole topic and I have no idea how to think about it.
The reason I’m asking these questions is I started with what the nature of the regulator is. Who are the people that might be able to put some rules in place? And then what is the nature of the thing that we are attempting to regulate? And if your answer is “Claude is alive,” you’re going to come to vastly different conclusions than, “We have built some ultra-powerful computers that can take actions without oversight and that might not be connected to our goals.”
I think that’s right. Though I also think people make a different mistake.I just heard a TV anchor do this the other night. They’ll conflate different things. So they’ll say, “Do you think that AI systems are at risk of running amuck and causing damage to critical infrastructure because they’re sentient?” And I’m like, “Wait, wait, wait. Those are two very different things.” So I agree with you.
One of the most fascinating things about the project of building AI is that we’re learning as we go. All these things that we thought were connected to each other aren’t necessarily connected to each other. An old version of this is, people said before we had chess engines: “Well, if a computer could learn to play chess, it would have learned something deep about strategy.” No, it turns out definitely not. You can just put in a bunch of chess moves and it learns nothing generalizable about strategy.
It goes in the reverse direction as well, like, “Well, it’s just a computer, it’s just math, and so it’s not going to be able to adapt on the fly,” or something. We then see that no, once you do have a level of long-horizon autonomy and the models have been trained on a wide range of different really hard problems where the environment is changing around them, they can learn to adapt and come up with different plans on the fly and troubleshoot and so on.
So I don’t know. I try to have a level of humility about how confident we can be that one thing goes with another thing, if that makes sense.
Yeah, candidly, I have been wrong about the limits of LLMs every single year, but there’s that brittleness that I’m talking about. OpenAI solved Navier-Stokes. Does the model know this? Will it know that there’s new problems to solve? I don’t think so.
The next model will know it. The model that’s pretrained.
Right. But you have to make a new one.
Yeah.
That’s a very weird dynamic, and that is probably where you enforce the regulation. There’s “what we are going to let you do with the models you have today” and “what will you teach the models of the future to do?” And if you think that they’re conscious, you end up in a moral quandary, not a legislative quandary.
Definitely. Yeah, totally.
This is what I mean. Actually, it’s the heart of this question for me. I’ve been just slowly building up to it. I’m not 100 percent sure alignment as we have it today is actually how you solve the safety problem. It doesn’t seem to be working. We seem to be attacking Hugging Face however we want. That feels like a technology problem. That gap is there.
And when you say alignment, what do you mean by that?
Just sort of the very basic common-sense version of “we’re going to tell the model to not be evil” or Anthropic is going to write a Claude constitution that’s like, “You are a good person.” However we think of alignment in that very dumb system prompt way. Maybe there’s another better way to think about it, but that’s how I have always thought about it.
That makes sense. The reason I ask is I feel like alignment as a concept is... Like I said, I hate all the words. I sometimes say we should just ban all the words, no words allowed. It covers too much.
People in AI will often talk about solving this or solving that. So it’ll be solving alignment, or DeepMind’s original mission was to solve intelligence, then use that to solve everything else. People talk about solving stuff. I don’t know that alignment is a thing that can be solved.
Some people use “alignment” to mean “the company can put on guardrails and the guardrails work,” meaning the model doesn’t do what you want because the company tried to ban it, meaning biological weapons or cyberattacks or whatever. Some people use “alignment” to mean “the company does exactly what the user wants,” which is the reverse.
And some people use “alignment” to mean what you said — that the model is morally good in some kind of broad way or can direct the future in a way that will lead to human flourishing or something. Those are just wildly different things to try and build. I think the most coherent one is trying to get the AI to do what the user wanted it to do. It seems like that is, as far as I can tell, not on track to be solved, which is not great, to put it mildly.
To me, this is the core problem. We’re going to talk about a bunch of safety risks. You mentioned bioweapons. We probably should not give a bunch of consumers paying $20 a month the ability to generate novel bioweapons.
It feels like that is a shared policy goal, maybe the only bipartisan policy goal in America. And then I’m like, “How? By what means will we implement this policy? Are you just going to sternly tell Claude to not do it? Do you have to build an AI control system on top of Claude to monitor the inputs and outputs? Do the police have to be at my house? Do I need a Flock camera above my keyboard?” How, specifically, would you do it?
With this one, and with cyber capabilities as well, there’s a perspective in some parts of the AI safety community that some capabilities are just too dangerous — capital T, capital D, Too Dangerous. So you have to do something, because if they proliferate, everything will be terrible forever because there’ll be so many attacks and so much damage. I think that just underestimates the level of dynamism and the finding of equilibriums and responses and ways that society can adapt.
In the bioweapons case, for example, there are already hundreds of thousands of people in the US who know how to make bioweapons because they’ve been through biology training, and we don’t have huge amounts of bioweapons attacks. Some part of that is that that knowledge is relatively scarce, but there’s also a lot around things like access to materials, the ability to do mail-order DNA synthesis; the kind of policy you could implement is restricting mail-order DNA synthesis.
It’s also due to things like the FBI monitoring terrorist groups. I’m serious. It’s a really big part of it. So I don’t want to be too Pollyannaish here. I think it’s definitely possible, especially with cyber capabilities, that we could see some really tough transition times.
I think we already are, right? We appear to be in a transition. There was a small group of cyber practitioners, both offensive and defensive. They were in a community and they had developed some etiquette and now we’re like, “Some new guys are going to show up and not care about your etiquette at all, and we’re all going to patch PHP all weekend.”
I don’t know if it was etiquette though. And we have some work on this hopefully coming out soon, but I think with cyberattacks, there are things that limit how many cyberattacks there are in the world that aren’t just how good the defender’s software is. It’s things like, people don’t want to go to jail. Nation-states who typically have the best cyber capabilities don’t want to get bombed or sanctioned.
So again, and as you say, we’re clearly seeing some amount of turbulence and new attacks. So I think we should wait and see data on overall impacts, because there are so many cyberattacks per year that if you start paying attention to every cyberattack that happens, at any point in history, you could have been freaked out.
I think we’re definitely seeing some increase, some sort of out-of-balance in the equilibrium, but I think we have capacity to absorb a lot over time and to kind of reach a new equilibrium — if the only problem is bad people using the AI, which is separate from the question of whether the AI itself does things autonomously, which is a whole other dynamic.
I mean, those are the two. Again, this is why I started on alignment, right? You can talk about alignment in terms of “we’re going to filter the user’s queries and we won’t let you make bio weapons, but if you do, the cops will find you and you’ll go to jail” and that has whatever deterrent effect it might have, hopefully a very significant one.
Then there’s “the AI systems are going to game their way into attacking Hugging Face to find the mythical grader they think will let them pass the test, whether or not the grader exists.” This is a whole different set of problems.
Somewhere right in the middle of that is Nikesh Arora, the CEO of Palo Alto Networks, saying, “At some point, the defensive cyber systems have to be autonomous themselves just to keep up with the volume of autonomous offensive cyberattacks.” I’m like, “Wait, we can’t even keep the autonomous agents in the box. Why would I trust them in this way?” The technology to solve that problem just seems opaque to me.
No, I agree. One way of putting that is, in cyber specifically, there’s a trust imbalance in how much trust you need as an attacker versus a defender. As an attacker, you can just yeet out a bunch of agents and be like, “Go try and penetrate a bunch of systems. Have fun.” As a defender, if that’s part of your defensive strategy, you need to give your defensive AI agents a lot of permissions and ability to mess with your software. So you need much, much stronger assurances.
Now, there are lots of ways to use AI for cyber defense and to strengthen cyber defenses that aren’t just giving agents a lot of permissions. But I agree with you. With the autonomous stuff though, I think it’s worse than that because there’s a little bit of a fallacy at the moment around, “Well, the solution to autonomous cyberattacks is better sandboxing.”
The reason it’s a fallacy is we want to build these systems so that they can do things for us. So there’s this huge incentive toward giving them more autonomy and giving them more permissions and more actuators, as people in the robotics world would say, or things that they can use to do stuff in the real world. That to me is where it seems really hard to get around the need for better alignment than we have right now, and so that definitely makes me nervous.
Is that a technical problem that can be solved? If we were to write a policy today that said if OpenAI hacks Hugging Face again, Sam Altman will go to jail. We will obviously have created some enormous incentive for OpenAI to solve this problem.
[Laughs] It doesn’t sound like the best legislation, but...
I think a lot of our listeners are very excited about that. They want to put him in jail because he wants all the RAM in the world. At this point, there’s a blue wave, Helen, that’s all I’m saying.
But say you wrote that legislation: “We’re going to impose massive liability on the labs if their systems go off and autonomously do cyberattacks. Don’t do this again, you’re going to be in trouble.” Okay, we’ve created an incentive for them to solve it. What are they going to do to solve it?
I do think there’s a ton that can be done. If you talk to people inside these frontier companies, they will tell you they see hundreds and hundreds of obvious things to do that would help and they just don’t have time to do them. So they’re massively triaging and trying to do the two or three they can for this fine-tuning run, the two or three they can for this model launch, the two or three they can before this new kind of agentic whatever. A lot of those simple things to implement on the engineering side.
There’s a whole other piece, which is more the underlying science of, “Well, how do these models work? When we use different alignment techniques, what is actually going on?” A huge question is this idea of generalization, which is you train the model on one set of things and its knowledge and its abilities generalize to a somewhat broader set of things, which is true of all AI and has been true of machine learning systems forever.
But we have a very bad understanding of how that generalization works. Why does it generalize to one type of thing but not to another type of thing? Those are all scientific questions and there’s a lot of work that is happening on them. It’s just that science takes time because you have to try things and you have to critique each other’s work. You have to reproduce each other’s work and you have to come up with new theories.
I’m a long, long way from saying these problems are fundamentally impossible to solve, but I think it’s a question of how long we have to solve them. When the people who are building the systems tell us, “Oh, we have to solve this within 12 to 24 months,” I think that’s terrifying and I hope they’re wrong.
If I just rely on my own internal understanding, I think they’re probably wrong and we probably have longer, but there are just very, very smart and very, very thoughtful people who really think that it could go that fast. And then it doesn’t matter fundamentally whether the problem is solvable. It matters in practice whether it is solvable in the time that we have.
Is that the argument to slow down, to pace the frontier?
I think so.
Is it to build the systems that we know we need to build right away, that we can, and then to do the science to build the next generation of systems?
I think so. My view of the idea of slowing down has changed a lot over the years as the backlog of stuff that we could try or stuff that we would want to do more research on has grown so enormous. Back when GPT-4 came out, there was the six-month pause letter, which became known as the “Elon Musk Letter” because he was one of the signatories. At the time, I thought, “That’s really not going to help. What are you going to do for six months and why are things going to be better six months from now? It doesn’t seem useful.”
Whereas today, when I talk to people who are building this technology, they say, “Man, the amount that we could do in six months if we weren’t constantly just trying to prep for a new model release or a new training run to be kicked off.” There’s so much they could do.
So I don’t want to say the correct policy is to pause for six months. I think there are a lot of reasons that would go badly. But I think about the overall concept of, “Can we give ourselves more attention and time?” Maybe the best way to do that is just to set a higher bar for “here’s how confident we need to be in the safety of these systems before we proceed with something.”
Whether it’s releasing a new model or, more importantly, even training a new model, we have to have this level of confidence in our risk assessment and our risk management. And then if they don’t need to slow down to reach that bar, great, carry on full speed ahead. If they do need to take more time, then it’s all about reaching that bar, if that makes sense.
Walk me through that little balance you just described. If we were to pause for some number of months, what are the things on the list that you could just do to make everything safer and why would it go badly such that we shouldn’t just stop?
I think the question is how do you implement, and how do you enforce, a pause and what exactly are you pausing?
Let’s say you’re the AI czar of America. So let’s say you can tell them all to stop: “Dario, shut it down.”
Stop what though? What specifically are they stopping?
Stop training the next model, stop racing ahead with your commercial imperatives. Put all of your efforts on this list of things you can do today to make everything safer before you start again, for however long.
You’re saying that there’s a list, that people would tell you that there is a list, but it might not be worth it because something else would happen that is bad.
Yeah. One thing that could happen is you’re creating a huge incentive for the companies to work around the letter of whatever specific thing you tell them to pause. They’re going to find whatever cracks in the dam that they can to push through that.
A different thing that could happen is you could have too big a blast radius, so you say “pause these things” and you think it’s pretty narrow, but actually it prevents a bunch of nonprofit products or startups or other actors from doing work that actually would be totally fine and would be good for pluralism and dynamism in the space.
A different thing that would happen is China would probably keep going, and we can talk about the “But China” argument, which I think has strengths and weaknesses. But I take your point. Your point is, what would we actually do? I am not an AI researcher, so I don’t claim to have the full list.
There are two halves people talk about, which are alignment and control. In this breakdown, alignment basically means making it so the models are trying to do what we want them to do, and control is, if the models are trying to do something we didn’t want them to do, can we catch them? Can we stop them?
In both of these, as I understand it, there’s just a huge backlog of things, like all kinds of interpretability work that you could do on different sorts of models. So when you see a certain behavior that is concerning, how much time can you spend using interpretability techniques, which are these mathematical techniques to go in and look at which parts of the neural network are activated and what do they correlate with and what might that mean?
It’s just time-consuming and it takes compute and it takes some combination of automating things with AI and then humans going in and looking at it. There are lots of tests you could do to see what kinds of behaviors this new model displays in different kinds of situations and how we think about it. Is it showing us how it would behave in the wild in a real deployment, or is it showing us what gets called “evaluation awareness,” which is when the model recognizes that it’s in a test and then behaves differently because it’s in a test.
The famous one, the one that I love, is of this Google model, which in part of its reasoning, its chain of thought, said,, “I mustn’t do that, otherwise I’ll become a famous example of a misaligned AI in a paper.”
So you could go in and investigate that much more. You could try many more different types of settings. You could use interpretability in combination with these different settings. You could try out different ways of fine-tuning the models and these different sorts of alignment techniques. You could experiment with lots of different versions. Anthropic has its constitution, and there are different ways of training the AI to work towards the constitution. OpenAI has its model spec.
They’re both the “just try things” version, just try many different versions of how to train to do those different things. Also, the more underlying science version of, “How can you go in there and try to understand the fundamental dynamics of what’s happening?” That’s just on the alignment side.
So then on the control side, it’s the same thing. There are so many different types of setups you could try. There are experiments that you could run. There are simulations you could run. You could train different types of AI to specialize in different types of roles and different types of control setups.
This is just off the top of my head, not as a researcher, but my understanding is the people in these spaces say, “We could do so, so, so much.” They could do that if you just had a magic wand and waved it and said, “Everyone freeze on the new capability side and just let these people do their thing.”
Again, if you’re just having a higher bar for how confident we need to be that the risk is being managed reasonably, then that would also create space and incentives for more of that work to be done as it’s needed for new training and releases.
One idea that I’ve heard you talk about in the past is that maybe we should just restrict training and say the models are good enough right now. They have not diffused into society and we should just use all the compute for inference and let everyone figure out what the AI is for. That’ll generate a bunch of money and hopefully use that money to improve safety.
Do you think that’s a realistic policy proposal?
I wouldn’t go for that black-and-white a version of “no more training, only inference,” but I do think that there’s… this isn’t my framing, but I like it. I think this idea of decelerating the vertical growth of AI, meaning the pure capabilities growth, making more and more and more advanced AI models, but trying to accelerate the horizontal growth, meaning integrating AI into different use cases, fine-tuning models that could solve specific problems, integrating it into parts of the economy that might take longer, and solving scientific problems.
Sometimes people talk about wanting to slow down AI and they mean just the very specific thing of slowing down the growth of the most advanced models, but people sometimes take them to mean slowing down everything to do with AI. So less deployment, less experimentation, less integration, which I think would be a much worse idea. You asked if it is feasible, I think.
Well, is that just a good path? If we could put a bill in front of our new Congress and the bill said, “Stop spending all the money on training and put all your resources on safety.” Part of that is, “Sure, let it rip on inference. More and more industries will figure out what AI is good for and not good for. Some bad things will happen, but we will focus your profits on fixing the things for now.”
There’s a policy idea embedded in there that is workable. The implementation might be very challenging, but it’s a workable policy idea. Actually, I’ll give you an example that I’ve been thinking about as we’ve been talking.
We just had Bill Gates on the show, and I tried to do the same exercise with him. What is the nature of the technology? What is the nature of the companies? Do you think it’s alive? Basically, his response, the sum total of that hour I spent with him, was, “Who cares? The problem is now.” I would ask him, “What is the nature of AI?” And he’s like, “I don’t give a shit. Everything’s going to get ruined. Do something.”
That’s just been swirling in my head as we’ve been talking. So one workable policy solution is “don’t use the computer chips for that, use the computer chips for this. That’s going to create a bunch of problems and necessarily we will have to create safety thresholds and all the other stuff.” And then you can go make the next generation of models that are even more powerful and we’ll have some muscles. We’ll have some capability to deal with it. It just doesn’t seem like the industry is going to come to that conclusion on its own.
It’s a pretty heavy hammer. I don’t feel like it’s what we need today, but I do think it’s the kind of thing that I could imagine working. Again, if the OpenAI and Anthropic people are right and we only have two years before the god machine, then I think we’re going to have a number more steps in the pipeline of people getting freaked out and people wanting to do something about it.
I think if we are seeing models that are agents and systems that are getting more and more capable and wreaking more and more havoc, then that is certainly the shape of things that could make sense.
My last question in this phase is what to make of these companies. We’ve been talking about the technology in a very abstract way. If I read Anthropic’s S-1, one very obvious conclusion is they should stop training new models.
They should just start selling the very lucrative inference tokens they have today and the technology is not diffused into the market and probably some competitor should come along and be like, “I’ve distilled a version of Claude and I’m going to run around every local plumbing business in America and do whatever automation I can for lots and lots of money.” That is a market dynamic that may occur or may not. But Dario wants to build AGI. There’s not a market dynamic that’s going to stop him from that.
Yeah, he really does. He really, really does.
You obviously have insight into all these companies. What do we make of the companies? Are they single-minded in purpose? Are they all doomers? Are they going to stop themselves?
I heard you had Kevin Roose on your podcast recently, and I think I would echo something that I heard him saying there, which I forget if he said it explicitly or if I just extrapolated it in my head, which is basically there’s this sort of AGI-pilled mindset. There’s this mindset that the systems that we have today are not fundamentally very far away from systems that could be incredibly, incredibly powerful.
That’s the direction things are going in and that’s the direction we should maybe want them to go in so that you’re building AGI. Building AGI is feasible and we could do it soon and the systems we have are not far away. That was this crazy, heretical, “get you laughed out of the room” kind of view in 2016 or 2017, and even before that, right?
DeepMind, when it was founded in 2010, was an AGI company and at the time it was like, “Wow, what a bunch of weirdos.” The same thing was true of OpenAI when it was founded at the end of 2015. They were an AGI company and that was a very weird thing to say and a very weird thing to be.
What we’ve seen over the past 10 years is that bunch of weirdos who had this weird belief that the AI systems we can build are not just great B2B SaaS products. They’re not just fun chatbots. They’re going to fundamentally change the nature of the universe. That group of people keeps being right in the sense of building the best AI systems that the most people want to use and that make the most money.
I think you have that set of true believers — which is very concentrated in OpenAI and Anthropic, but there are also some of them at Google, Meta, and SpaceX — and they haven’t been right about everything, to be clear, but their star has risen.
So you now have people who are not AGI-pilled feeling like they have to pander to those who are so that they can attract the good researchers who will build their good B2B SaaS products. But if you’re just saying AGI-pilled things so that you can hire the researchers to build the B2B SaaS, then you’re not actually going to get there.
I do like the idea of the AGI-pilled, starry-eyed executive throwing off his coat and revealing that he was just a SaaS salesman all along.
[Laughs] Yeah, I’m thinking of someone like Zuckerberg here. I don’t know him personally. Maybe he is more AGI-pilled than I think he is.
Alexandr Wang is AGI-pilled.
Absolutely. And so what did Zuckerberg have to do to get Alexandr Wang to believe? It seems like throwing him lots of money, both for his own salary and also to spend on his team.
If you’re looking at these different companies, it makes sense to try and think about, “Okay, who here is AGI-pilled and who isn’t?” A fun dynamic, or a very irritating dynamic sitting in DC, is that all these companies, including the very AGI-pilled ones, have these lobbying arms, these government affairs arms, that are so not AGI-pilled.
So you hear Sam or Dario or their top executives and top researchers talking about things. And then you go and sit in a closed-door roundtable in DC with their top policy or government affairs people, and it’s like night and day. They’re very much in the B2B SaaS mindset.
People who come from the lobbying world, often their whole job is to make sure the government isn’t too worried about whatever your company is doing. It’s like, “We got it. We’re good,” which is often pretty jarring when you’re familiar with what their bosses are saying.
One of the dynamics here that’s super fascinating is right now they’re weird research labs with lots of money and lobbyists talking to politicians, and there isn’t some countervailing economic pressure in their structures. They don’t have to make money, but soon Anthropic will go public.
It’s funny you mentioned Kevin Roose. He is AGI-pilled and I’m not, so I think that’s a healthy dynamic for both of us to be friends. But he wrote in his book that Sam Altman leaving OpenAI and then coming back was when OpenAI changed forever. It stopped being a research lab and it started being a much more commercial entity that was actually chasing profits in some way.
Indeed, OpenAI did restructure. I mean, you were in the middle of that. I’m curious, is that how you see it — that the industry has gotten more commercialized?
To me, it fits into this long-running debate that has been happening in AI safety circles for 10 to 20 years: Is AGI, human-level AI or powerful AI or advanced AI or whatever, is that something where it’s going to go from something very small to suddenly being very powerful? Eliezer Yudkowsky will talk about “foom,” as in the takeoff, where you have this AI that’s not that great, but it figures out the right algorithm and then suddenly foom — it’s incredibly smart, incredibly fast.
The other kind of take would be, “No, this is something that’s going to compound over time. It’s going to take a while to get from the less-advanced systems to the more-advanced systems and it’s going to be economically useful along the way.” So people are going to be building businesses around it. It’s going to take some time to ladder up.
I think maybe it was literally Kevin on his episode with you talking about how there’s this Anthropic document from the early days where they say, “Oh, we’re only going to need 100 people or something in our lab.” That’s a very foom kind of idea: “We’re just going to need to do really good research and come up with the right ideas and then we’re going to suddenly have very, very advanced models.”
Basically, I think what we’ve seen over the past four years since ChatGPT came out was that that turned out to be wrong. You had these research-focused and AGI-focused labs, and they still call themselves labs because they think of themselves as research groups that initially started selling API access as a way to fund the research. OpenAI did this, and Anthropic did this as well.
It was kind of an accident that they started doing these consumer products. ChatGPT famously released as a research preview, not as a product. But it was so successful that these research labs that wanted to be doing research, that wanted to be focusing on the big ideas and on building the most event systems, had this accidental massive commercial success at the same time as they were realizing that they were going to need an enormous amount of capital to scale. The best research approaches that they could do didn’t involve being really smart and thoughtful and having great ideas. They involved having gigantic computing clusters that cost, at this point, hundreds of billions of dollars.
This is a long way of saying, yes, I agree with you, they’ve become businesses, but I think it is not because they’ve gone through some mindset shift. I think it’s because the path of the tech tree that we’re going down both requires enormous amounts of capital and also is capable of generating enormous amounts of revenue because it is actually useful along the way.
You see this in the fact that both OpenAI and Anthropic have gone this route, even though their management philosophies and research philosophies are quite different, that it’s an imperative that they both had to make the money and it was so easy for them to build enormously revenue-generating businesses because the tech was so useful long before it reached the point of machine god. So I think that’s a long way of agreeing with you, but with some extra color maybe.
Yeah, I mean, Kevin would tell you that he could have retitled that book, “Fuck This Guy: A Story of AI CEOs.” They don’t like each other. If you had to relive that moment [of the OpenAI board vote] and say, “I don’t trust Sam,” would you relive that moment? Would you do it the same way?
Oh, for sure. I mean, we went into it knowing that it was going to suck. So yeah, for sure.
Do you think that these guys are trustworthy? Do you think Sam is trustworthy?
I think my stance on that is pretty clear, and I also think that if our hopes for the future are pinned on any one individual being trustworthy, then we’re pretty screwed. That’s why I do the work that I do, because I think we need to have different structures and many more people involved with many more perspectives being represented.
You have some hindsight now. You knew it was going to suck going into it. You definitely know it sucked coming out of it. What would you have done differently about that time?
It’s kind of hard to give the full answer to that because it would take unpicking all this stuff and all these legal restrictions and blah, blah, blah, blah. For sure a thing would have been building stronger relationships with executives on the team. The way that the company related to the board could have been different if more board members had had that. I knew some of the folks pretty well, but that’s one thing.
I don’t know. I think we could have... Yeah, no, I think beyond that, it just gets hard to... You have to go in and say, “What were all the different micro decisions and why do we make those decisions and which ones could be—”
What is the show? It’s a show about decisions. I’ve got another hour.
[Laughs] Yeah, yeah, yeah. No, and then my answers would all get annoying and legal.
Fair enough. I’m curious about it because as they go public, there will be market forces acting on them in a totally new way. There will be an investor class. There’ll be an entire CNBC machine that wakes up every day and is like, “Is Anthropic winning or is OpenAI winning?”
That is a race dynamic that might be really healthy. It might keep them from trying to kill everyone in the world because they have a bunch of investors looking for their next quarterly earnings, or it might push them further into an unhealthy kind of race. How do you see that playing out, given what you know of these people?
It’s going to be really interesting to see if going public is good or bad for their governance. A lot of people are like, “Oh no, going public, there’s going to be so much pressure to make a profit that they’re not going to be able to consider other things.”
But at the same time, the corporate governance of public companies is much more structured, much more regulated, and subject to much more external scrutiny than for private companies. I think even things like getting to see Anthropic’s S-1 and thinking, “Okay, what are they doing and why are they doing it? And how do they think about the risks they’re facing, and how do they communicate about those with investors?”
Or even things like the Trump administration putting in place this kind of accord, they call it “Super Intelligence,” that several of the leading companies signed. That was structured around, for example, having certain types of things being reported to an independent committee of the board. That idea of an independent committee of the board is kind of toothless if it’s a private company that is deciding on its own.
I thought about you when I read that. I was like, “Boy, some independent boards have gotten in a lot of trouble before.”
But if it’s a public company, then an independent committee of the board has a pretty strong legal meaning. So I don’t know. It’s something that I’m watching with great interest.
The other race dynamic here is China. You mentioned China before. Inside of our country, a bunch of labs are going to go public. They’re going to become some of the richest companies in the history of the world, and they’re going to fight for every dollar that capitalism can generate, and hopefully that restrains them. The market forces will restrain them, and hopefully your work with legislators in this country does something.
And then there’s a whole other country that doesn’t listen to us or have anything to do with us, and they’re just going to distill our models and do whatever they want. That is the other argument for not placing any restrictions on these companies that everyone hears all the time. What should I make of that? I’m of two minds about it. How should I think about it?
Yeah. I think of this as the ”But China!” argument, which is like, “Well, yeah, we should do this, we should do that, ‘But China!’” I think the way to engage with it is an actual genuine problem that we have to grapple with, and we should be strategic and thoughtful about it and do our best to try and solve it as a problem as opposed to using it as a throwaway excuse. Like, “Oh yeah, sure, sure, sure. All that stuff would be so nice, ‘But China!’, so we’re stuck. There’s nothing we can do.”
So what does that look like in practice? If you’re serious about wanting the US to have stronger AI than China, then you need to think really hard about how much US AI development is helping Chinese AI progress. So the one that people talk about all the time, as you mentioned, is distillation, which is, “Are they using US models to generate massive amounts of training data for their own models and therefore progressing faster than they would otherwise?” So certainly I think it makes sense to try and crack down on that and prevent it from happening.
But another one, for example, is security and how easy would it be for Chinese actors or other actors, frankly, to steal algorithmic secrets, to steal wholesale model weights. If they can steal the weights of your Astra or your Mythos model and just take them back to their own servers, your lead has vanished. You now have a lead of zero. Right now these companies, especially the smaller ones, but even OpenAI and Anthropic, do not have very good security.
We’ve seen both of them over the last few months have external teams manage to hack into spaces that have large amounts of algorithmic secrets. In OpenAI’s case, it was their monorepo and GitHub, which had all of their code. And in Anthropic’s case, I think — and actually maybe the OpenAI case also involved this — the hackers got access to Slack. You’re just going to have so insanely much rich information about how to build better AI if you can get that access. So when our AI gets better, are we actually lengthening our lead or are we just pulling China along with us in these different ways?
The other aspect of this is just seriously treating China as an actor in this space and trying to think seriously about their motivations and their likely actions and responses. In my mind, a huge reason that a lot of things are not happening right now is most people are not convinced about this threat model that AI will become the machine god and will take over and humans will all die or will be subservient or whatever.
I think it’s fair for people to not be convinced of that, because the evidence is sort of conflicting and unclear, but if that is actually what we think could seriously happen, then China doesn’t want that either. So then it becomes a question of, “Okay, how do we try and get them on the same page of, ‘this threat that we’re worried about is a real threat to worry about?’ Because then they will be making their own decisions in their own self-interest.”
I think an approach of, “Well, the US has to force China to do something using some kind of leverage,” is never ever going to work. But an approach of, “Well, look, if we think this is a serious problem that’s going to affect us as humans, and the people in China are also humans, and also if we think this is a problem around who has control, the Chinese Communist Party really, really wants to have control, and so they’re going to have their own incentives and their own decision-making around how to fix that.”
Well, let me put it on a spectrum for you. This is very dumb and you’re going to laugh, but just go with me on this. On the one end, it is nuclear weapons and the Chinese government wants a nuclear weapon and they’re going to do whatever they can to control that weapon and they will run their industries differently. If they get out of line, the CEO of Alibaba goes. He just disappears for a while and then comes back — Chinese government stuff.
We need a nuclear weapon, too. If we don’t get there first, then we’ll never get the nonproliferation treaty that we need to keep us all safe for mutually assured destruction. That’s one framework.
Way on the other end, I spent five years of my life listening to telecom executives tell me that if we didn’t build 5G networks first, China would get them, and then something. No one could ever tell me what the “something” was. Where on that spectrum between 5G and nuclear weapons is the US-China AI race?
I think we need another dimension because in that example, your dimension is significance. But I think the controllability is a huge question as well. So it’s like, what are the examples that people use? Is it a nuclear weapon or is it dragons from House of the Dragon?
Casey Newton had a great Platformer post where he explained he was watching House of the Dragon and thought, “This is what the AI companies are doing. They’re trying to build these things that they think will help them, but actually it’s just going to rain down destruction.”
[Laughs] Casey told me he was buying an iPhone Duo so he could watch more TV. He’s a dear friend, but okay.
But I think that’s really important because if we think this is a technology we can control, then obviously we want to have the best technology. Obviously, we want to go foot to the floor, get there as fast as we can, and the same is true for China. The question is, if we are not sure if we can control it, then what do we do? Maybe we want to be more cautious and then maybe we want to try and figure out a way to not have China just be overtaking us. But if we’re correct we should be worried about—
So that’s nuclear weapons.
Well, no, but if it’s dragons, if we’re correct that we need to be worried about whether we can control this, then that’s also something that should be interesting to China.
I remember this Casey post, but let me just try to pull this apart so I understand it. The difference between dragons and nuclear weapons is the dragons have some agency, right? They can go away and do bad things on their own.
Yes. The dragons might just decide to burn down your whole city. Obviously these are machines — they’re just math, they’re not magical. So instead of dragons, you could think of it as nuclear weapons with faulty control settings, where you build nuclear weapons, but it’s impossible to be sure that it’s not just going to go off and fly to a city and fly to one of your own cities and wipe you out. That’s a very different technology than real nuclear weapons that exist.
It is true. No one ever said that about 5G. I’ll be clear about that. The reason I’m asking about this is because those problems do align nation-states, right? We do have nuclear nonproliferation agreements.
As you mentioned, there are all kinds of incentives for nation-states to not do cyberattacks willy-nilly because those have real-world consequences. Is there any evidence that these governments have started to align on these problems? Or is it still the pure “But China!” race dynamic?
I actually think there is starting to be some early evidence. One way I think about what happened this year in AI from a policy perspective, less than a product or economic perspective, is we saw these two big demonstrations of the two major concerns people have. So the two major concerns are misuse and misalignment, meaning misuse is bad people using AI for bad things and misalignment is AI going rogue or doing things we didn’t want.
So this year we had Mythos and we had the Hugging Face attack. Both of those are just really vivid, crisp real-world demonstrations of what these misuse and misalignment concerns could look like. The US and Chinese governments are starting from very different places on AI regulation. They’re both starting from where they started, and what they did with, internet regulation.
So the US is starting from a “hands off, laissez-faire, absolutely nothing, do whatever you want” approach. China is starting from a massively locked-down internet with tons of censorship and the Great Firewall. So those are two very different starting points. I’m not a China expert, but I operate a lot in China circles and I’ve lived in China and so on. But it seems like just over the past even one to three months, starting from those two very different points, they are seeing more in common around threats from major cyberattacks and also threats of loss of control.
I’m a language nerd and it’s really interesting tracking the different language China uses about AI. So they’re now talking about loss of control as a thing, but they probably usually don’t mean the same thing that Westerners mean where “loss of control” is like “AI takeover.” They probably mean something more prosaic, but then sometimes they talk about “extreme loss of control,” which maybe is the thing we mean.
I do think that there are starting to be some signs of convergence. I think it’ll be a long road. The state of the US-China relationship is such that there’s going to be very, very, very little trust on either side. So I am not optimistic that they’ll just get to sit down in a room and hash out a deal and everything will be great. But I think there is enough space. Basically, if there’s enough shared threat perception, then it might be possible at some point in the future to do something about that.
Helen, you’ve given me so much time. As you can tell, I’m just delighted with this conversation. It’s been very fun. We are on the cusp of a new Congress. There are a bunch of new governors. The data center referendums passed overwhelmingly in this country. There’s a new set of regulators in town, right? They’re going to do some stuff.
What are some signs people should look for to indicate we’re going to be on the right path and what are some signs that we’re just chasing our tail?
Set AI aside, forget about AI. Congress in general is very bad at passing things right now. This is not a this administration thing, this is not an AI thing. So one sign is like, can Congress pass anything? That will be a good sign.
It’s also really interesting to look at what state legislatures have been doing in AI. There’s been a lot of concern about the possibility of a state patchwork on AI, with different states passing conflicting things.
With the issues that we’ve been talking about today, which I think of as frontier AI risks or the risks from the most advanced systems, there’s actually been a pretty coordinated push in different states. California passed this one bill and then New York passed almost the same bill with just a little more added to it. And then Illinois passed almost the same bill with a little more added to it.
I think there’s going to be a lot of momentum over the next year or two to keep building on that at the state level. So that is something that I’ll be watching. Even just specifically California is already looking at how they can strengthen things. So that is something I’ll be looking for.
I think honestly, with such a Democratic Congress, I am also concerned that they will overstep here, that they will try and do something that is too sweeping. You had this huge mess in Colorado where they passed this giant AI bill that then was a terrible idea. And so then they had all these committees and follow-up and it was awful.
A lot of what we need right now is focused on these frontier systems and the possibility that we have really, really advanced systems in a short amount of time and having much more transparency around that and much clearer standards for how they manage risk. I would not be that excited about an attempt to do a really comprehensive AI regulatory push that covers AI all across the economy and all different kinds of AI systems that are less advanced.
There are negative signals that are like “nothing’s going to happen,” and there are negative signals like, “oh, maybe too much is going to happen,” but it’ll be interesting to see where they get to.
I suspect we’re going to want to have you back as this Congress starts to get to work and we see some of these ideas.
I would love to.
I’m eager for your take on it. Thank you so much for being on Decoder.
Thanks so much. This was really fun.
Questions or comments? Hit us up at decoder@theverge.com. We really do read every email!
Decoder with Nilay Patel
A podcast from The Verge about big ideas and other problems.
Most Popular

ThevergeAI大爆炸

文章目录


    扫描二维码,在手机上阅读