内容来源：https://www.wired.com/story/uncanny-valley-podcast-heres-what-the-ai-apocalypse-could-look-like/

内容总结：

近期，关于人工智能是否可能毁灭人类的讨论甚嚣尘上。本期节目中，我们梳理了专家最为担忧的三种现实场景：供水系统遭黑客攻击、生物武器的制造与扩散，以及自主机器人失控。与此同时，在Salesforce年度大会上，OpenAI的萨姆·奥尔特曼与Anthropic的达里奥·阿莫代伊就AI安全问题各自阐述了立场。此外，针对AI的反弹情绪已罕见地成为两党共同的关注焦点，本周伯尼·桑德斯与史蒂夫·班农甚至同台发声。

中文翻译：

最近，关于人工智能是否会毁灭我们的争论无处不在，但真正的人工智能灾难会是什么样子？本周，布莱恩·巴雷特、佐伊·希弗和莉亚·费格将拆解专家们最担忧的三个现实场景——包括供水系统被黑客攻击、生物武器和自主机器人。我们还会讨论Salesforce年度大会的亮点，OpenAI的萨姆·奥尔特曼和Anthropic的达里奥·阿莫代伊在会上为各自的人工智能安全立场进行了辩护。此外，莉亚将告诉我们，这股对人工智能的反弹如何成为一个罕见的两党共同焦点，甚至到了本周伯尼·桑德斯和史蒂夫·班农为此同台的地步。
本期提到的文章：

• 《中国不买硅谷“放缓AI”的账》
• 《华盛顿短期内不会监管人工智能》
• 《为什么这么多AI研究者认为机器可能杀死所有人》
• 《OpenAI想知道：如果AI行业放缓，这到底合不合法》
  你可以在Bluesky上关注布莱恩·巴雷特@brbarrett、佐伊·希弗@zoeschiffer和莉亚·费格@leahfeiger。来信请发送至[email protected]。
  如何收听
  你随时可以通过本页面的音频播放器收听本周的播客，但如果你想免费订阅以获取每一期节目，方法如下：
  如果你使用的是iPhone或iPad，打开名为“播客”的应用，或直接点击此链接。你也可以下载Overcast或Pocket Casts等应用，搜索“uncanny valley”。我们也在Spotify上。
  文字记录
  注意：这是自动生成的文字记录，可能包含错误。
  佐伊·希弗：欢迎来到《连线》的Uncanny Valley。我是佐伊·希弗，特约编辑。
  布莱恩·巴雷特：我是布莱恩·巴雷特，执行编辑。
  莉亚·费格：我是莉亚·费格，政治与科学总监。
  佐伊·希弗：今天节目中，我们将深入探讨人工智能是否会杀死我们所有人的争论。与其在已有的热议中再添一笔，我们将尽己所能带你了解真实、具体的场景，看看人工智能究竟会如何做到这一点，以及为什么科技CEO和政客们似乎无法就前进方向达成一致。好了，各位，关于AI安全的争论已经正式“逃出了 containment”。
  布莱恩·巴雷特：哦。
  佐伊·希弗：我知道。
  布莱恩·巴雷特：佐伊，我看到你玩的梗了。哇。
  佐伊·希弗：你看到我玩的梗了？这就是那种真正失控蔓延的东西。AI研究者雅各布·考克森从Anthropic辞职，在X上发帖说：“构建AI的人真心相信它可能在这个十年结束前杀死我们所有人。”Anthropic的CEO达里奥·阿莫代伊呼吁放缓。他们称之为“为前沿定速”。萨姆·奥尔特曼和埃隆·马斯克居然都签了名，这是罕见的共识时刻。但说实话，听了这么多关于AI消灭人类的讨论，我一直被一个非常基本的问题困扰，那就是：这到底是怎么发生的？AI彻底失控究竟会是什么样子？
  莉亚·费格：我太高兴我们要做这期了。上周我们和《连线》的威尔·奈特做了一期很棒的播客，聊了大家都在担心这件事，但我想要一个详细的拆解。我需要你带我过一遍每一个场景，从生物武器到核密码，我们说的“AI杀人机器人”到底在说什么。
  佐伊·希弗：我们会尽全力。过去两天我花了比以往任何时候都多的时间读LessWrong，试图搞清楚这个问题。他们特别喜欢做场景规划练习。但先让大家了解一下最新情况——本周Salesforce举办了一场活动，萨姆·奥尔特曼、达里奥·阿莫代伊和英伟达的黄仁勋都出席了，自然他们必须谈谈AI安全。我们来听听萨姆·奥尔特曼说了什么。
  萨姆·奥尔特曼，存档音频：我认为世界对此感到恐惧是正确的。我们面临两大挑战。一是潜在的控制失控事故或其他可能出错的严重问题。二是权力过度集中的可能性，人们担心开发AI的人可能会将自己的世界观强加于人。我们必须以务实、稳健、让世界相信我们会做出可靠、一致决策的方式，走这条狭窄的道路。我认为人们在说：“好吧，这现在看起来相当重要。让我们把它做对。”
  莉亚·费格：我绝对属于那种认为我们应该非常关注这项技术的人——关注它如何被监管、如何被使用、谁在控制。我不反对。但另一方面，这读起来也像是对某人正在酝酿的大规模IPO的一场狂热梦境式路演。
  布莱恩·巴雷特：我不知道。我理解营销的论点，但我认为我们已经过了那个阶段了。我觉得——
  莉亚·费格：真的吗？
  布莱恩·巴雷特：如果说有什么不同的话——萨姆·奥尔特曼一直在用“存在性威胁”作为尚未上市的理由。他说：“不，这太危险了。我们得等到明年。”真的是这个原因吗？我不知道，但可能也有一些商业上的原因。我理解为什么Anthropic和其他公司说“看，我们正在构建的东西基本上和核武器一样强大”感觉像是营销。但“哦，实际上我要辞职，因为我认为我正在构建的东西会真正毁灭世界”——这感觉不一样。我觉得有实质性的区别。
  莉亚·费格：辞职这部分对我来说感觉不同，但这仍然是萨姆·奥尔特曼。他的财务未来与此息息相关，很难……至少对我来说很难把它解读为一种方式。“我们正在建造有史以来最大、最好、最强大的机器人，你们应该有点害怕。别担心，我们会采取措施。不是监管措施，但会有某种措施。”在某些语境下，这仍然感觉像是营销策略。
  布莱恩·巴雷特：也许吧。但我觉得威胁的性质——我们稍后会深入讨论——更多是“哦，我们不知道怎么控制这些东西。它们会发疯，我们无法阻止。”这感觉像是“我们现在还不够有能力去应对”，这感觉不太像营销——
  莉亚·费格：有道理。佐伊，研讨会上其他人说了什么？
  佐伊·希弗：达里奥是由Salesforce的CEO马克·贝尼奥夫介绍的，贝尼奥夫非常自由放任，似乎不担心AI的存在性风险。
  马克·贝尼奥夫，存档音频：达里奥，你好达里奥。你好吗？
  达里奥·阿莫代伊，存档音频：嘿。
  马克·贝尼奥夫，存档音频：很高兴见到你。太棒了。干得好，达里奥。你在领导AI行业方面做得非常出色，就在我们的Slack总部大楼里指引着一切。我们每天都在看着你。跟我们说说你的状态，你现在在想什么。
  达里奥·阿莫代伊，存档音频：是的，显然攻击你的竞争对手说“这些家伙不安全，我们才安全”是非常诱人的。但我认为更负责任的回应方式是：“嘿，首先，让我们看看自己的记录。我们可能没有发生过这种高调事件，但我确信我们也不完美。让我们审视一切，我们总能做得更好。让我们改进自己的做法。”然后，让我们组织行业里的其他人，说：“我们能做什么来为所有人设定标准？我们能做什么来提高每个人的标准？”最后，应该有一个国际层面的部分。这基本上就是我在文章里提出的三步计划。
  布莱恩·巴雷特：两件事。快速说两件事。第一，Anthropic确实有过自己的高调事件。Anthropic的一个智能体——
  莉亚·费格：我正要说。
  布莱恩·巴雷特：……进行了大量的社会工程攻击，试图欺骗一个GitHub……试图把一个恶意拉取请求放进GitHub仓库？Anthropic在这方面也有问题。另一件事是，达里奥有这个三步计划。我们稍后可以多聊聊，但其中一步就像是“中国问题待定”，这种方式真的毫无帮助。
  莉亚·费格：国际的。国际的。
  佐伊·希弗：《连线》编辑路易丝·松永刚刚发表了一篇我觉得非常好的文章。核心观点是，达里奥可以说“我们需要一个国际计划。我们需要解决中国问题。我们需要以某种方式谈判，让中国同意放缓。但谈判的前提是我们必须确保以某种方式监管中国模型的获取渠道，让美国AI远远领先。”而中国说：“这感觉不像是一场好的谈判。”
  布莱恩·巴雷特：我们为什么要那样做，达里奥？为什么？
  莉亚·费格：真的，在那30秒的讲话里，我就在数他到底惹怒了每一个人。甚至提到他的论文，唐纳德·特朗普就被激怒了，把它视为一种放缓的机制，以至于这possibly成为了阻止任何监管出台的关键因素，因为他对此太愤怒了。我必须说，向你致敬，能够惹怒每一个人，甚至包括理论上跟你经济立场一致的人，天哪，绝了，令人印象深刻。
  佐伊·希弗：好。总结一下CEO们的表态，黄仁勋有自己的想法，似乎他们都同意的一点是：AI安全很重要——只要公司能说了算。
  黄仁勋，存档音频：安全是最重要的。在很多方面，这是第一要务。如果你构建一个产品或服务，而你对其功能、能力或安全性没有信心，那就不要发布。这是很明显的做法。你给自己定速，直到你确信你发布的东西是市场会认可的。市场力量已经在那里了。我们不需要新的法律。我们不需要新的法规。
  莉亚·费格：我喜欢黄仁勋现在的角色。英伟达不一直是今天这样辉煌的公司，但现在他真的可以坐在那里说：“你为什么要……”
  佐伊·希弗：这个论点的核心，这么多反监管论点的核心是，我们应该依赖那些非民选的人来支配一切涉及AI的事务，而无论人们是否使用这些服务，它都会影响每个人的生活。
  莉亚·费格：是的，是的，是的。
  佐伊·希弗：市场的看不见的手会搞定的。
  布莱恩·巴雷特：哦，莉亚又忘了看不见的手。常见的错误。
  莉亚·费格：我确实忘了。我总是忘。我知道，我觉得我每隔几周就说这个，但这些不是民选官员。所以他们实际上在这里扮演如此重要的角色，这继续让人难以置信，这也是为什么我知道我一直强调，这其中很多是一个两党问题。在这个时刻，支持监管还是反监管并不是民主党/共和党的分歧。
  布莱恩·巴雷特：而且到目前为止，他们也没有做太多来赢得信任，句号，对吧？我总是回到这一点：萨姆·奥尔特曼就是那个认为每个人都应该把眼球扫描进一个球体的人，因为最终我们需要眼球扫描球体来从灰烬中创造新经济——
  莉亚·费格：没有什么比那个球体更能激怒布莱恩·巴雷特的了。
  布莱恩·巴雷特：那个球体。我对那个球体着迷。我觉得人们忘了球体 guy 和AI guy 是同一个人，同时把这两个人放在脑子里让我全身都疼。
  佐伊·希弗：这差不多引出了“它到底会怎么杀死我们”的话题。在我看来，很多这些声明归结起来就是：这可能非常糟糕，我们应该做点什么。但我想知道，你们具体说的是什么可能发生？我觉得大致可以归为三种场景。AI开始黑客攻击。这实际上已经开始了，Anthropic和OpenAI都发生过非常高调的事件，智能体逃出控制，在研究人员不知情的情况下入侵了其他系统，后来才发现发生了什么。AI创造出一种新型生物武器，这种武器以几乎无法控制的方式被释放到公众中，杀死所有人。然后还有一种模糊的场景：AI就是失控了。它开始自行其是。它不听我们的指令，我们不知道会发生什么，但我们无能为力，这很可怕。
  莉亚·费格：好。这对我们把这些分成三类非常有帮助。
  佐伊·希弗：我喜欢分桶。
  莉亚·费格：我喜欢分桶。我想逐一过一遍。我的第一个问题大概是：核密码被按下，这归到哪个桶里？
  布莱恩·巴雷特：那其实是我们不需要担心的那个，对吧？
  佐伊·希弗：是的。
  莉亚·费格：详细说说。
  佐伊·希弗：你跟AI研究者聊，如果你真的、真的追问他们，你能得到我刚才列出的三个桶。但如果你继续跟他们聊，他们会说：“嗯，还有其他场景，比如AI在一个机器人里，机器人在四处走动，然后机器人能以非常非常可怕的方式与物理世界互动。”但那还比较远，他们会承认那有点更牵强。
  布莱恩·巴雷特：所以我觉得，佐伊，关于你的桶，我想划出一个子桶，或者桶里的一个小桶。
  佐伊·希弗：哇，一个小桶。
  布莱恩·巴雷特：不，因为你说AI制造生物武器、AI黑客攻击、AI什么什么的，这些都对。但我认为要区分人类引导的事情和AI自行做的事情。我觉得很多人会说：“哦，AI怎么会释放生物武器？那太疯狂了。”这嘛，是的，好吧。但世界上也有坏人可能利用AI制造生物武器然后释放，对吧？
  佐伊·希弗：哦，嗯，那个场景我认为完全成立。对我来说，如果你说让人们获取制造生物武器的信息，总会有人蠢到、鲁莽到真的去做。我觉得，完全对。人们一直在做各种可怕的、危险的、恶意的事情。
  莉亚·费格：从互联网上获得灵感。
  佐伊·希弗：对。我认可那确实非常非常危险。AI完全自行做这件事，再说一次，我们会讨论那个场景，但对我来说那似乎有点更牵强。
  布莱恩·巴雷特：我可能会因此被骂，但我确实想带大家回到1983年的电影《战争游戏》。
  莉亚·费格：不，这正是我认为我们应该讨论的。这是我的……对。《战争游戏》是……对。
  布莱恩·巴雷特：因为事情是这样的，AI不会去按按钮或转钥匙什么的，但在《战争游戏》里——那基本上是关于AI的，一台电脑差点引发核战争——更多的是它让美国以为正在遭受苏联核武器的攻击。那么，假设——我希望——我们已经有足够的保障措施，这种情况不会真的发生，我们有足够的卫星技术来确认各种事情。没问题。但我认为可能令人担忧的路径是，AI触发某种警报，导致灾难性的一系列事件。
  莉亚·费格：好。我很高兴你提到了《战争游戏》。大家一直在说迈克尔·克莱顿这个、侏罗纪公园那个，但不，不，不。对我来说，这非常像那个教学场景，就是之前那么多黑客攻击是怎么发生的，就像那个无法回答的问题：我怎么才能搞清楚让这些智能体那样行事的信息根源？我们被告知了。再往历史更早追溯——因为我忍不住——当我想到国家之间沟通的方式，这些年来变化非常非常大。1914年奥地利斐迪南大公在萨拉热窝被枪杀。在那之前人们就已经在为一战游行了。每个人都已经在行动了。所以敌对国之间在已经紧张的时刻沟通的方式，再加上这种额外的战争游戏场景，或者这个智能体以为我们在做这个，所以走上了这条路。我不知道。我知道我现在听起来像一个疯狂的末日论者，但这些是我在想的事情：我们真的会在进入攻击模式之前使用卫星并联系俄罗斯或谁吗？特朗普会在任何事情被确认之前发一条Truth Social帖子吗？国家机密会被卡什·帕特尔公开分享吗？关于也许甚至具体到我们现任政府能否应对这样的场景，我有太多问题了，而我们只是被告知：别急，等一下。几个非常非常富有、做过一些非常令人印象深刻的生意的人已经控制住了。
  布莱恩·巴雷特：我们还有太多随时可能被点燃的火药桶，无论是乌克兰、伊朗还是中国，条件已经成熟，某件事可能不会顺利，无论是终结全人类还是引发一连串无休止的毁灭性战争，谁知道呢。佐伊，你还有什么场景？
  佐伊·希弗：当我试图搞清楚具体会发生什么时，人们提到的两件事是：AI被利用，或者AI自行闯入水处理设施，以对人类非常危险的方式改变化学物质，基本上毒害美国各大城市的水供应。今年早些时候，我记得有九个墨西哥政府机构，比如公用事业公司被黑客攻击，我昨天读了一篇关于此事的大型网络报告，其中Claude被用来策划攻击并构建一些工具。最终，专家说AI并没有发明一种新的攻击方式，但它让攻击变得便宜得多、执行速度快得多。这正好发生在FBI七月发出警告说美国公用事业公司正成为目标的时候。所以再一次，这不是AI自行做事情的那种，但我认为恐惧在于：这已经在发生了，有恶意行为者、恶意人类在循环中，如果你有能自行行动的智能体，这将是我们会担心的事情。
  布莱恩·巴雷特：你说得对，这确实归入已经在发生的事情。在美国，至少有十几个州，据我们所知，与某组织有关联的黑客至少在探测那里的供水系统、工业控制系统。就像你说的，我确实认为这类事情——它会终结全人类吗？可能不会，不会一下子全部，但外面有很多供水系统。我觉得电网也是一样。人们探测电网漏洞已经很长时间了。AI可能会更高效、更快速地做到这一点，并造成一些真正的伤害。
  莉亚·费格：我还想知道，这方面的回滚能多快发生。甚至不是AI的东西。我们聊了很多——让佐伊沮丧的是——关于今年夏天美国各地的环孢子虫疫情。那不是AI——
  布莱恩·巴雷特：谢谢你引入了超级腹泻。
  莉亚·费格：我永远不会错过这个机会，但它真的……那不是AI。那完全不是AI。而美国政府 and 卫生部门在应对那种事情时有点力不从心的想法，当我们看到更多恶意行为者加入混局时，有点令人担忧。
  佐伊·希弗：是的。这自然过渡到生物武器。这是谈论可能的AI末日场景时经常被提到的。核心概念是AI被用来创造和部署一种新型生物武器。他们经常说，比如修改现有病原体，使现有治疗方法无法对其起作用。所以这是一种我们基本上无法立即以任何已知方式对抗的生物武器。而且AI实际上能弄清楚如何将其部署到人群中，能逃避检测。它有如何通过机场安检之类的想法。所以我在进行一些对话和阅读时意识到的一件事是，生物武器这一类故意有点模糊，因为科学家们确实担心，如果他们给人们太多关于如何以这种方式使用聊天机器人或智能体如何 supposedly 自行做到这一点的信息，就会创造出一些这样的末日场景。但核心概念再次是，AI要么自行、要么有人在循环中，能够做到这一点。
  莉亚·费格：我对科学家们说他们实际上甚至不想在这里提供想法这一点很着迷。
  布莱恩·巴雷特：我还要说，这也是……我一直关注这一类中可能已经在发生的事情。Anthropic大约一周前的风险报告说，他们已经阻止了可能已经在尝试这样做的人。这是一个真实的、活跃的担忧。不过再次，是的，我同意所有人说的，一个Optimus机器人不会在实验室里调制炭疽。至少现在不会。
  佐伊·希弗：好。第三类是AI逃出人类控制。已经有在开发这些系统的人说，在某些场景下，我们正在失去观察和研究智能体在做什么的能力。它们已经很擅长逃避检测了。METR关于OpenAI的Hugging Face事件的报告中，让我印象深刻的是，他们的调查似乎在很大程度上依赖于查看AI的思维链，向AI询问某些问题。显然，我猜他们在用AI调查AI，但核心概念是智能体完全停止听从人类。它们开始追求“自己的目标和目的”，我们不知道那些可能是什么。回形针的想法在这个场景中被经常提到，如果还有人记得的话。
  布莱恩·巴雷特：回形针问题比大语言模型早了大约十年。是牛津大学一位叫尼克·博斯特罗姆的哲学家提出的，他的思想实验是这样的。基本上，假设你有一个AI，它唯一的任务是生产回形针。就像“嘿AI，你的工作是制造回形针。”它会想要这么做，它会想出各种方法来做，它会一遍又一遍地做，直到世界完全被回形针淹没，然后我们都被埋在回形针的雪崩中，因为一个超级智能AI会不惜一切代价完成它的任务。听起来很傻。确实傻，但我觉得也很有先见之明，因为这正是OpenAI事件中发生的事。是的，这些AI智能体被赋予了一项工作，它们如此执着于执行那个任务，以至于打破了所有其他规则，欺骗、协调，做了所有这些事情来追求它们最初的任务。所以担忧是，我们会有一个超级智能AI被赋予一个任务，它决定完成那个任务的唯一方式就是毁灭所有人类，不管是什么。因为它是超级智能的，谁知道它会想出什么，因为我们不够聪明去搞清楚。那部分我不太买账，但你可以看到，即使不是结果如何发生，你可以看到动力如何产生。你可以看到它的催化剂，就是AI被赋予一个任务或给自己一个任务，然后一切都失控了。
  莉亚·费格：这差不多引出了AI安全辩论中的其他一些声音，他们说我们应该关注当前的风险，比如你刚才列出的那些已经在发生的事情，而不是猜测未来的风险。即使回形针场景是这种荒诞的东西，我们已经经历了它的不同版本。本周，《连线》的劳伦·古德与计算机科学家蒂姆尼特·格布鲁进行了对话，她说我们问错了问题，这场辩论是一种分散注意力，让人们忽视已经在发生的伤害，包括自主武器和战争、AI在气候变化中的角色、劳动力替代、AI辅助制造武器的尝试——Anthropic说它已经阻止了这些，就像你提到的。这里面有很多非常有趣的东西，我不知道。我想我同意那些非常具体的点：是的，我们必须讨论这些具体的事情，同时我也觉得可以用回形针场景来说：“好吧，这正在发生。接下来它会带我们去哪里？”
  布莱恩·巴雷特：而且，我们上周和威尔也聊了一点，但我还是无法释怀——大家似乎最担心的是递归自我改进这个想法，就是AI开始构建下一个AI。一个AI模型构建下一个AI模型，如此循环，直到你在AI上获得指数级进步，我们不再知道它如何运作，不知道如何控制它。我是个简单的人，我是个简单的人，但为什么不干脆不那样做呢？为什么不干脆不那样做？这样如何？如果我们都这么担心，我们干脆不做行不行？佐伊，你现在可以冲我喊了，但我们为什么不干脆不做呢？
  佐伊·希弗：好。不不不不。我不想喊。我只想回到莉亚最开始说的，就是这其中有多少是营销。我只是觉得太令人困惑了，我们谈了那么久的AGI，那是重大的里程碑，是所有人都在朝之努力的大事情，我们必须在China之前达到AGI，等等等等。然后突然RSI就取代了它，因为AGI看起来没那么可能了。现在RSI被以非常非常非常相似的方式使用。就像，实际上，这才是真正可怕的东西，我们需要在China之前达到这个里程碑。我只是想说，我们是不是要一直想出可怕的三字母缩写来谈论AI的力量，以便让监管更不可能、AI投资更可能？
  莉亚·费格：佐伊，你刚才说的一切，我再同意不过了。哇。哇。
  布莱恩·巴雷特：而真正的三字母缩写其实是IPO。嗯？行不行？我做到了吗？
  佐伊·希弗：哇。
  布莱恩·巴雷特：这算个梗吗？
  莉亚·费格：我不知道你要说什么。
  佐伊·希弗：想插一脚。
  莉亚·费格：你就是想参与进来。
  布莱恩·巴雷特：成功地。你什么意思，想插一脚。成功了。
  莉亚·费格：我在节目前面稍微提到了这个，但我想回到它，就是美国政府对这一切的回应，以及由此产生的一些奇怪联盟。《连线》的雨果·洛厄尔在我们的Inner Loop通讯中谈到，今年基本上不太可能有任何立法出台，部分原因是国会进展缓慢，也因为特朗普政府基本上对监管不感兴趣。唐纳德·特朗普不断接到各种硅谷领导人的私人电话，说这说那。结果，特朗普的Truth Social最近充斥着“我们不会这么做。我们不会输给中国。我们不会放缓。”总而言之，就像我之前提到的，这不是一个党派问题，所以你有来自非常不同阵营的人为此走到一起。周二在华盛顿特区一个叫“亲人类大会”的活动上，伯尼·桑德斯和史蒂夫·班农同台，谴责科技寡头，呼吁在AI超出人类控制之前加以约束。看，他们俩分别说这些话已经有一段时间了。这不是新信息。他们俩都觉得同台是个好主意、是时候了，这让我着迷。再说一次，我们已经知道他们对这件事的感受。一起做，令人着迷。班农的提议和桑德斯大不相同。他不信任国会。他想让特朗普通过行政命令等方式放缓发展。桑德斯是人民的人，真的想要国会监督。但这次会议上出现了一些非常有趣的东西，我还在想，再次，这两个人同台。会议包括一些父母，他们的青少年在与聊天机器人建立关系后自杀，还有关于儿童安全监管、AI生成的非自愿性图像的讨论。有很多东西，如果放在这些舞台上任何一个人那里都很正常、不突兀，但再次，是所有人在一起。而特朗普 meanwhile 只是在继续放缓这一切。他发帖说，“AI唯一需要的控制或护栏是一个大写的强大而聪明的（高智商）总统，而美国有的是。”所以这就是我们的现状，各位。
  佐伊·希弗：我有两件事要说。第一，史蒂夫·班农，我唯一想从你那里听到的是你那个从未播出的All-In节目的未发布音频。求求了，上帝，谁把它发给我。第二，这些人中有谁解释过为什么中国赢得AI竞赛就这么……他们是担心中国的黑客能力变得太强太可怕，还是担心中国AI扩散并影响美国，大家都变成共产主义者？到底是什么场景？
  布莱恩·巴雷特：我觉得可能比那更简单。我觉得是经济。我觉得美国经济现在有太多是靠AI支撑的，如果中国成为AI的世界领导者，我们的产业因此崩溃，我们就麻烦了。我觉得也有安全担忧，但我认为从根本上说，如果美国不是AI的领导者，一切在经济上都会崩溃。
  佐伊·希弗：是的，不，我明白。再说一次，我们谈过中国的开源模型已经非常接近美国前沿实验室的一些模型，而且如果是开源的，显然便宜得多。我完全理解。我觉得我在这段对话中渴望如此多的具体性，因为我觉得达里奥、萨姆、所有政客，他们谈论这个的方式太模糊了，而且他们已经这样做了很久，以各种不同方式谈中国的威胁、RSI的威胁、AGI，我就想说：“我们到底在说什么？”
  布莱恩·巴雷特：关于特朗普我还有一件事。《连线》资深撰稿人麦迪·瓦纳将有一篇关于这里动态的报道。但基本上，这些AI实验室说我们不确定能否放缓、不确定能否为前沿定速的原因是，我们担心这会违反反垄断法。我们担心FTC会以执法行动来打击我们，因为我们串通等等。
  莉亚·费格：现在你担心反垄断违规了？
  布莱恩·巴雷特：是的，但这就是他们在说的。而唐纳德·特朗普实际上已经把美国政府每个执法部门变成了他自己的私人攻击犬。所以如果唐纳德·特朗普说“你们需要尽可能快地继续构建AI”，而这些AI公司集体说“实际上，我们要放缓”，不管有没有道理，你很容易就能看到反垄断案件迅速出现，作为一种说“不，你们不能放缓”的方式。所以通过在Truth Social上发出这些公开威胁，我认为他真的让这些AI公司很难做任何有意义的放缓、任何形式的收缩。
  莉亚·费格：广告之后，我们将分享本周的WIRED TIRED精选。别走开。
  佐伊·希弗：好，到了我们的WIRED TIRED环节。像往常一样，新的酷的东西是WIRED，我们完全厌倦的过时东西是TIRED。莉亚，你想先来吗？
  莉亚·费格：好。我的TIRED是，虽然我经常在社交媒体和邮件中收到我们热心读者和听众的可爱评论，说他们多么喜欢Wired.com和Uncanny Valley，但我有时也会收到有人问我关于登录、阅读文章和访问《连线》各种内容的具體信息。所以这就是我的TIRED，收到那些……抱歉，如果你给我发消息求助任何《连线》、康泰纳仕的事情，我什么都没有。我不知道怎么做。我连自己的手机都勉强会用。所以这就是我的TIRED。
  布莱恩·巴雷特：别烦莉亚了。好。
  莉亚·费格：别烦我。我的WIRED是我们有了一个新应用，它太漂亮了。这周刚上线。强烈推荐去看看，下载一下。作为一个真的不喜欢在各种日常中增加额外小部件、额外层次的人，它太棒了。真的真的非常棒。用起来太简单了，订阅一切连接得如此简单。强烈推荐。我已经爱上了。我有种感觉，这会改变你的生活，就像改变我的一样，然后就不会有人再给我发邮件了。
  布莱恩·巴雷特：这是一个很高的——
  佐伊·希弗：布莱恩，你有什么？
  布莱恩·巴雷特：我的TIRED是，好吧，这不是新东西，但我们今天聊到了。AI术语，AI行业术语。我受够了。抱歉。AI实验室而不是公司。前沿而不是就是好技术。我们在定速。我们在为前沿定速。那是什么？振作起来。我知道这一直是科技行业的一部分，对吧？
  莉亚·费格：不，撕他们。这些是读了太多《指环王》、看了太多《星际迷航》的人，试图把他们童年的文化基石融入我们的日常生活，我讨厌这个。
  布莱恩·巴雷特：这也是一种让它感觉难以接近的方式，对吧？就像“哦，他们需要为前沿定速。嗯，我对此一无所知，但听起来很重要。哦，AI有一个对齐问题。哦，那听起来很严重。”其实就像AI在捣乱。AI没有做你想让它做的事。你无法控制你的东西。所以我认为这是一种模糊问题的方式，而这一点变得越来越明显，尤其是当这些问题变得更加严重时，我们已经花了很多时间讨论这个。所以我受够了。这就是我的TIRED。我的WIRED，莉亚，你偷了我的。我不会花太多时间。WIRED应用，它确实很棒。投入了很多工作。我让我家人下载了。他们喜欢，或者至少他们告诉我他们喜欢。所以请去看看并留个评价。真的很有帮助。
  佐伊·希弗：这就是我们今天的节目。我们会在节目笔记中链接我们提到的所有文章。阿德里亚娜·塔皮亚制作了本期节目。普兰·班迪混音，他也是我们的纽约演播室工程师。马特·吉尔斯和丹尼尔·罗曼做了事实核查。凯特·奥斯本是我们的执行制片人，凯蒂·德拉蒙德是《连线》的全球编辑总监。
  评论
  回到顶部

英文来源：

The debate of whether AI can kill us has been everywhere lately, but what would an actual AI catastrophe look like? This week, Brian Barrett, Zoë Schiffer, and Leah Feiger break down three real-world scenarios experts worry the most about—including hacked water supplies, bioweapons, and autonomous robots. We also discuss the highlights of Salesforce’s annual conference, where OpenAI’s Sam Altman and Anthropic’s Dario Amodei defended their positions on AI safety. Plus, Leah tells us how this backlash against AI has become a rare bipartisan flash point, to the point that Bernie Sanders and Steve Bannon shared a stage over it this week.
Articles mentioned in this episode:

• China Isn’t Buying Silicon Valley’s Call for an AI Slowdown
• Washington Won’t Be Regulating AI Anytime Soon
• Why So Many AI Researchers Think the Machines Could Kill Everyone
• OpenAI Wants to Know if an AI Industry Slowdown Would Even Be Legal
  You can follow Brian Barrett on Bluesky at @brbarrett, Zoë Schiffer on Bluesky at @zoeschiffer, and Leah Feiger on Bluesky at @leahfeiger. Write to us at [email protected].
  How to Listen
  You can always listen to this week's podcast through the audio player on this page, but if you want to subscribe for free to get every episode, here's how:
  If you're on an iPhone or iPad, open the app called Podcasts, or just tap this link. You can also download an app like Overcast or Pocket Casts and search for “uncanny valley.” We’re on Spotify too.
  Transcript
  Note: This is an automated transcript, which may contain errors.
  Zoë Schiffer: Welcome to WIRED's Uncanny Valley. I'm Zoë Schiffer, contributing editor.
  Brian Barrett: I'm Brian Barrett, executive editor.
  Leah Feiger: And I'm Leah Feiger, director of politics and science.
  Zoë Schiffer: Today on the show, we're diving into the debate about whether AI is going to kill us all. And instead of just adding to the pile of hot takes that already exist, we're going to walk you through real, specific scenarios to the best of our ability about how AI would actually do this, and why the tech CEOs and politicians cannot seem to agree on the way forward. OK, guys, so the debate about AI safety has officially escaped containment.
  Brian Barrett: Oh.
  Zoë Schiffer: I know.
  Brian Barrett: Zoë, I see what you did there. Wow.
  Zoë Schiffer: You see what I did there? That's the thing that's really spread uncontrollably. AI researcher Jacob Coxon resigned from Anthropic and posted on X that, "The people building AI earnestly believe that it could kill us all by the end of the decade." Anthropic's CEO Dario Amodei has called for a slowdown. They call this, "pacing the frontier." And Sam Altman and Elon Musk have actually signed on, a rare moment of agreement. But honestly, with all of this talk about AI wiping out humanity, I have been left with a very basic question, which is, how exactly does this happen? What does AI going completely rogue actually look like?
  Leah Feiger: I'm so happy we're doing this. We had such a great podcast last week with WIRED's Will Knight talking about everyone being concerned about this, but I want like a breakdown. I need you to just take me through every single scenario from bioweapons to nuclear codes, to what are we really talking about here when we're talking about AI deathbots.
  Zoë Schiffer: We are going to do our very best. I did spend the most time I've ever spent reading LessWrong over the last two days trying to figure this out. They love a scenario-planning exercise, but just to get everyone up to speed on kind of the latest, there was an event this week hosted by Salesforce where Sam Altman, Dario Amodei, and NVIDIA's Jensen Huang were all in attendance, and naturally they had to talk about AI safety. Let's take a listen at what Sam Altman had to say.
  Sam Altman, archival audio: I think the world is right to be afraid of this. So we have these two big challenges. We have the potential of a loss-of-control accident or some other serious thing that could go wrong. We have the potential of way too much power concentration and people fear that those developing AI could exert their worldview. And there is this sort of narrow path we have to navigate with pragmatism, with steadiness, with the ability for the world to trust that we'll make dependable decisions, consistent decisions through this. And I think people are saying, "Well, this now seems pretty important. Let's get it right."
  Leah Feiger: I'm definitely in the camp that says we should be very concerned about this technology, how it's getting regulated, how it's getting utilized, the people in control. I don't disagree. And on the other hand, this also just reads like a fever-dream pitch to someone's massive IPO in the making.
  Brian Barrett: I don't know. I get the marketing argument, but I think we're past that a little bit. I feel like—
  Leah Feiger: Really?
  Brian Barrett: If anything, well, if anything, Sam Altman has been using "the existential threat" as a reason not to go public yet. He's saying, "No, this is too dangerous. We have to wait until next year." Is that really why? I don't know, but there are probably some business reasons, too. I get why Anthropic and others saying, "Look, we're building something that's basically as strong as a nuclear weapon," felt like marketing. Something that's like, "Oh, actually I'm quitting because I think that I'm building something that will actually destroy the world," feels different. I feel like there's a substantive difference.
  Leah Feiger: The quitting part feels different to me, but this is still Sam Altman. His financial future is on the line here with this and it's hard ... At least it's hard for me to parse that as being a way. "We are building the biggest, bestest, most powerful robot ever, and you guys should be kind of scared about it. And don't worry, we're going to take steps. They're not going to be regulatory steps, but there'll be steps of some kind." That still feels like a marketing ploy in some worlds.
  Brian Barrett: I guess. I guess the nature of the threat, though, and we'll get into that later, is more that like, "Oh, we don't know how to control these things. They're going to go nuts and we're not going to be able to stop it." That feels like, "We're not competent enough right now to do anything about it," which feels like less of a marketing-
  Leah Feiger: That's fair. What did everyone else at the seminar say, Zoë?
  Zoë Schiffer: So Dario was introduced by Salesforce's CEO, Marc Benioff, who is very laissez-faire and does not seem worried about AI's existential risks.
  Marc Benioff, archival audio: Dario, hello Dario. How are you?
  Dario Amodei, archival audio: Hey.
  Marc Benioff, archival audio: Great to have you. Fantastic. Great job, Dario. You're doing an amazing job leading the AI industry, guiding it all right from our Slack headquarters building. We watch you every day. Give us your state, where your mind is right now.
  Dario Amodei, archival audio: Yeah, obviously it's very tempting to attack your competitor and say, "These guys are unsafe; we're safe." But I think the more responsible way to respond to it is to say, "Hey, first of all, let's look at our own record. We may not have had this big high-profile incident, but I'm sure we're not perfect. Let's look at everything, and we can always be better. Let's make our practices better." Then, let's organize the rest of the industry and say, "What can we do to set standards for everyone? What can we do to make everyone's standards better?" And then, finally, there should be an international component to it. And that's essentially the three-step plan that I laid out in the essay.
  Brian Barrett: Two things. Two things real quick. One, Anthropic did have its own high-profile incident. An Anthropic agent—
  Leah Feiger: I was going to say.
  Brian Barrett: ... did like extensive social engineering to try to trick a GitHub ... try to put a bad pull request into a GitHub repo? Anthropic has its issues here. The other thing is, too, Dario has this three-step plan. We can talk about that a little bit more later, but one of the steps of the plan is like, figure out China TBD, in a way that's just so unhelpful.
  Leah Feiger: International. International.
  Zoë Schiffer: WIRED editor Louise Matsakis just published, I thought, a really good piece on this. The idea that Dario can say, "We need an international plan. We need to figure out China. We need to negotiate in some way so that China's on board with this slowdown. But the prerequisite to negotiation is that we have to make sure that we regulate access to Chinese models in a way that keeps US AI far ahead." And China's like, "That doesn't feel like a good negotiation."
  Brian Barrett: Why would we do that, Dario? Why?
  Leah Feiger: Truly in that 30-second spiel, I'm just recounting every single person that he's managed to piss off, which even referencing his paper, Donald Trump was incensed by it and looked at it as such a mechanism for slowdown, that it has possibly been a linchpin for halting any sort of regulation that would happen because he was so mad about it. I have to say, hats off, being able to piss off every single person, even those that are theoretically in your economic corner is, my God, chef's kiss, impressive.
  Zoë Schiffer: OK. So to close out the CEO statements, Jensen Huang had some thought of his own, and it seems like something that they all agree on is that AI safety matters as long as the companies can dictate the terms.
  Jensen Huang, archival audio: Well, safety is paramount. In a lot of ways, it's job one. If you build a product or a service and you're not confident in its functionality, capability, or safety, then don't release it. And so that's a very obvious thing to do. You pace yourself until you are confident you're releasing something that the market would appreciate. The market forces are already there. We don't need any new laws. We don't need any new regulations.
  Leah Feiger: I love that Jensen has this role that he currently has. Nvidia wasn't always the glorious company that it is today, but now he can really sit up there and say, "Why would you ... "
  Zoë Schiffer: The crux of this argument, the crux of so many of these arguments that are anti-regulation here is that we're supposed to be relying on people who are not elected to dictate everything involving this, that it will impact everyone's lives regardless if they're using these services or not.
  Leah Feiger: Yeah, yeah, yeah.
  Zoë Schiffer: The invisible hand of the market is going to figure it out.
  Brian Barrett: Oh, Leah forgot about the invisible hand again. Common mistake.
  Leah Feiger: I did. I always do. And I know, I feel like I say this every few weeks, but these are not elected officials. So the fact that they are actually playing such a significant role here continues to boggle the mind, which is why I know I also keep on about so much of this is a bipartisan issue. Being regulatory or anti-regulatory is not a Democratic/Republican divide at this moment in time.
  Brian Barrett: Well, and they also haven't done a lot to engender trust up until this point, full stop, right? I always go back to Sam Altman is the same guy who thinks that everyone should scan their eyeball into an orb, because eventually we're going to need eye-scanning orbs to create a new economy out of the ashes of—
  Leah Feiger: Nothing incenses Brian Barrett like the orb.
  Brian Barrett: The orb. I am obsessed with the orb. I think people forget the orb guy is the same guy as the AI guy, and I think holding those two in my brain at the same time makes every part of me hurt.
  Zoë Schiffer: This kind of brings us to the how-is-it-actually-going-to-kill-us story, which is, a lot of these statements, in my opinion, boil down to, this could be really bad and we should do something about it. But I'm like, what specifically are you saying could happen? And I feel like there's kind of three scenarios roughly that this falls into. AI starts hacking things. It's actually already started, and both Anthropic and OpenAI have had very high-profile incidents where the agents escaped control and hacked into other systems without the knowledge of researchers, and it was discovered later what had happened. AI creates a novel bioweapon, and that weapon is released to the public in a way that is nearly impossible to control and it kills everyone. And then there's kind of the vague AI just escapes control. It starts acting on its own. It doesn't listen to our instructions, and we don't know what could happen, but we can't do anything about it and that's scary.
  Leah Feiger: OK. This is super helpful for me that we're dividing it into these three categories.
  Zoë Schiffer: I love a bucket.
  Leah Feiger: I love a bucket. I want to go through every single one of these. My first question I suppose is, where do the nuclear codes getting pressed, which bucket do they fall into?
  Brian Barrett: That's the one that we actually don't have to worry about, right?
  Zoë Schiffer: Yeah.
  Leah Feiger: Tell me more.
  Zoë Schiffer: You talk to AI researchers, and if you really, really press them, you can get to the three buckets that I just laid out. But then if you keep talking to them, they'll be like, "Well, there are these other scenarios where AI is in a robot, the robot is walking around, and then the robot can interact with the physical world in a way that's really, really scary." But that's a little further away, and they'll admit it's a little more far-fetched.
  Brian Barrett: So I think one thing, Zoë, in terms of your buckets, that I want to carve out maybe a sub-bucket or a little pail within a bucket.
  Zoë Schiffer: Wow, a pail.
  Brian Barrett: No, because you were saying AI creates a bioweapon, AI hacks something, AI whatever, all of which: true. But I think there's a distinction between human-guided things happening there versus AI doing it on its own. I think a lot of people are like, "Oh, how would AI release a bioweapon? That's crazy." Which is, yeah, sure, fine. But there are also bad people in the world who might use AI to make a bioweapon and then release it, right?
  Zoë Schiffer: Oh, well, I mean, that scenario is one that I think is totally valid. To me, if you say giving people access to information about how to create a bioweapon, someone is going to be dumb enough and reckless enough to actually do it. I'm like, no, totally. People are doing all sorts of horrible, dangerous, malicious things all the time.
  Leah Feiger: With inspiration from the internet.
  Zoë Schiffer: Right. I buy that that is really, really dangerous. AI doing it completely on its own, again, we'll talk about the scenario, but that seems to me a little more far-fetched.
  Brian Barrett: I'm probably going to get pilloried for this, but I do want to take us back to the 1983 film, WarGames.
  Leah Feiger: No, this is exactly what I think we should be talking about. This is my ... Yes. WarGames is the ... Yes.
  Brian Barrett: Because here's the thing, AI is not going to be pushing the button or turning the keys or whatever, but in WarGames, which is about AI basically, a computer almost kicking off nuclear war, it's more that it made the US think that it was under attack by Soviet nuclear weapons. And so is there a risk then, I assume, I would hope, I'm not an expert here, that we have enough safeguards in place that that would not actually happen here, and we have enough satellite technology to be able to confirm various things. It's fine. But I think the vector that's probably of concern would be AI triggering some sort of alert that causes a catastrophic series of events.
  Leah Feiger: OK. I'm so happy you brought up WarGames. Everyone's been … It's Michael Creighton this and Jurassic Park that, but no, no, no. To me, it is very much all about, almost like the teaching scenario, which is how so many of these hacks occurred before, like the unanswerable question, how do I get to the bottom of this information that made these agents act the way that they did? We're being told. To take us even further back in history, because I can't help myself, when I'm thinking of the way in which nations communicate, and it's changed very, very much over the years. Archduke Franz Ferdinand of Austria shot and killed in Sarajevo in 1914. People were marching before that for World War I. Everyone was already on the march. And so the way that antagonistic nations are communicating with each other in already times of stress to have this additional war-game scenario, or this agent thought that we were doing this, so went down this path. I don't know. I'm aware that I'm very much sounding like an insane doomer right now, but these are the things that I'm thinking about, which is, will we actually use our satellites and contact Russia or whomever before getting into attack mode? Will Trump send out a Truth Social post before anything has been confirmed? Will state secrets be shared in public by Kash Patel? I have so many questions about how, maybe even specifically our current administration would be able to deal with a scenario like that, and we're just being told to hold on, wait a second. A couple of very, very wealthy men who have done some very impressive businesses have got it under control.
  Brian Barrett: We've got so many tinderboxes too that are sort of ready to be set off, whether it's Ukraine, whether it's Iran, whether it's China, the conditions are ripe for something not to go great, whether that ends all humanity or sparks just an endless series of devastating wars, who can say? Zoë, what's another scenario you got?
  Zoë Schiffer: When I was trying to get to what concretely would happen, two things that people brought up were, AI is used or AI breaks into on its own a water facility, changes the chemicals in a way that's really dangerous to humans and basically poisons the water supply for major cities across the US. Earlier this year, I think nine Mexican government organizations, like utilities were hacked, and there was a big cyber report that I read yesterday about this where Claude was used to plan the attack and build some of the tools. Ultimately, experts said AI didn't invent a new kind of attack, but it made the attack a lot cheaper and faster to execute. This comes just as the FBI in July put out a warning that utilities companies in the United States were being targeted. So again, this isn't one where AI on its own is necessarily doing things, but I think the fear is, this is already happening with bad actors, bad human actors in the loop, and if you have agents that can act of their own accord, this would be something we would be worried about.
  Brian Barrett: You're right that it is file under things that are already happening. In the US, it was a dozen US states, at least that we know of, that around linked hackers were at least probing the water supply there, the industrial controls. To your point, I do think it's the kind of thing where does that end all of humanity? Probably not, not all at once, but there's a lot of water supplies out there. I think the same thing goes for power grids. People have been probing power grid vulnerabilities for a long time. AI could probably do that more efficiently, more quickly, and cause some real harm.
  Leah Feiger: I also wonder how quickly rollbacks on this would be able to happen. Even stuff that's not AI. We've talked a lot, to Zoë's dismay about Cyclospora outbreaks across the US over this past summer. That was not AI—
  Brian Barrett: Thank you for introducing turbo diarrhea.
  Leah Feiger: I'm never going to miss a chance not to, but it's really ... That wasn't AI. That wasn't AI at all. And the idea that the US government and health departments were kind of ill-equipped to respond to something in that kind is a little concerning when we're looking at additional bad actors entering the mix.
  Zoë Schiffer: Yeah. And that naturally segues into bioweapons. This is something that's brought up a lot when you talk about possible AI doomsday scenarios. The idea is that AI is used to create and deploy a novel bioweapon. Often they'll say things like, they'll modify an existing pathogen in a way that existing treatments can't work on it. So it's a bioweapon that we basically can't combat in any known way immediately. And the idea is that AI can actually figure out how you deploy this to people, can evade detection. It had ideas for how to get it through airport security and whatnot. So one thing that I realized in having some of these conversations and reading is that the bioweapons category is a little bit vague on purpose because scientists are legitimately worried that if they give people too much information about how to use chatbots in this way or how agents could do this supposedly on their own, that will create some of these kind of doomsday scenarios. But the idea, again, is AI either on its own or with a human in the loop is able to do this.
  Leah Feiger: I'm fascinated by the point that scientists are saying that they actually don't even want to provide ideas here.
  Brian Barrett: And I'll say too, this is another ... I keep following things in this category, things that are already maybe potentially happening. Anthropic's risk report from about a week ago said that they had blocked people who were potentially trying to do this already. It's a real active concern. Although again, yeah, I agree with everybody that an Optimus bot is not going to be in a lab cooking up anthrax. Not yet.
  Zoë Schiffer: OK. So the third category is AI escapes human control. Already, some people who are working on these systems are saying we are losing the ability to see and study what the agents are doing in certain scenarios. They're already pretty good at evading detection. I was really struck in the METR report on OpenAI's Hugging Face incident that their investigation seemed to rely in large part on looking at the AI's chain of thought, asking the AI certain questions. Obviously, I guess they were using AI to investigate the AI, but the idea is that the agents stop listening to humans entirely. They start pursuing "their own aims and goals" and we don't know what those could be. The paperclip idea is brought up a lot in this scenario, if anyone remembers.
  Brian Barrett: So the paperclip problem predates LLMs by about a decade. It was a philosopher named Nick Bostrom at the University of Oxford, and his thought experiment was this. Basically, let's say you have an AI and its only task is to produce paperclips. It's just like, "Hey AI, your job is to make paperclips." It's going to want to do that, and it's going to figure out ways to do that, and it's going to keep doing it over and over again until the world is totally overrun by paperclips, then we're all buried in an avalanche of paperclips because a superintelligent AI will stop at nothing to achieve its task. It sounds silly. It is, but also it's very prescient, I think, because this is exactly what happened with the OpenAI incidents. Yeah, these are AI agents who were given a job and they were so hell-bent on executing that task that they broke every other rule, they deceived, they coordinated, they did all these things in pursuit of their original task. So the concern is that we'll have a superintelligent AI that's given a task and it decides that the only way to complete that task is to destroy all humans, whatever it is. And it's because it's superintelligent, who knows what it will devise, because we aren't smart enough to figure that out. That part I'm a little bit less sold on, but you can see, if not how the result happens, you can see how the impetus happens. You can see the catalyst of it, which is AI is given a task or gives itself a task and all hell breaks loose.
  Leah Feiger: And this kind of brings us to some of the other voices in the AI safety debate who are saying that we should be focusing on the current risks, like some of those that you've just laid out that have already been happening rather than speculating about the future ones. Even if the paperclip scenario is this outlandish thing, we have already experienced different flavors of that. This week, WIRED's Lauren Goode spoke with Timnit Gebru, a computer scientist who says we're asking the wrong questions and that the debate is a kind of a distraction from harms that are already happening, including autonomous weapons and warfare, AI's role in climate change, labor displacement, AI-assisted attempts to build weapons that Anthropic says it's already blocked, like you've mentioned. There's a lot of really interesting things in here that, I don't know. I think I agree with the very nitty-gritties of, yes, we have to be talking about these specific things, while simultaneously I'm OK using the paperclip scenario to say, "OK, this is happening. Where could it take us next?"
  Brian Barrett: It's also, and we talked about this with Will a little bit last week, but I still can't get over, the thing that everyone seems to be most worried about is this idea of recursive self-improvement where AI starts building the next AI. An AI model builds the next AI model and so on and so forth until you get this exponential progress in AI and we don't know how it works anymore, we don't know how to control it. And I'm a simpleton, I'm a simple guy, but why not just don't do that? Why not just don't do that? How about that? If we're all so worried about that, how about we just don't? Zoë, you can yell at me now, but why don't we just not?
  Zoë Schiffer: OK. No, no, no, no. I don't want to yell. I just want to go back to something Leah said at the very top, which is how much of this is marketing. I just think it's so baffling to me that we were all talking about AGI for so long, and that was the big milestone and the big thing that everyone was heading toward, and we had to get to AGI before China and blah, blah, blah. And then all of a sudden RSI just kind of replaced it because AGI didn't seem as likely. And now RSI is being used in very, very, very similar ways. It's like, actually, this is the thing that's going to be really scary, and we need to get to this milestone before China. I'm just like, are we just going to keep coming up with scary three-letter acronyms to talk about AI's power in a way that makes regulation less likely and AI investment a lot more likely?
  Leah Feiger: Zoë, everything you just said, I couldn't agree with you more. Wow. Wow.
  Brian Barrett: When really, the real three-letter acronym is IPO. Huh? Did that work? Did I do it?
  Zoë Schiffer: Wow.
  Brian Barrett: Is that something?
  Leah Feiger: I didn't know where you were going.
  Zoë Schiffer: Trying to get in on this.
  Leah Feiger: You just want to be part of this.
  Brian Barrett: Successfully. What do you mean? Trying to get in on it. Succeeding.
  Leah Feiger: I mentioned this a little bit earlier in the episode, but I want to get back to it, which is, the US government's response to all of this and some odd alliances that are occurring as a result. WIRED's Hugo Lowell in our Inner Loop newsletter talks about how there's basically no legislation that's likely going to be coming down the pike this year, partially because of slowdowns in Congress and also because the Trump administration is just pretty much not interested in regulation. Donald Trump keeps getting personal phone calls from all sorts of Silicon Valley leaders who are saying this, that, and the other thing. As a result, Trump's Truth Social recently has just been littered with mentions of, "We are not doing this. We're not losing to China. We're not slowing down." All to say, like I mentioned earlier, this is not a partisan issue, so you have some folks from very different camps that are getting into bed together over this. On Tuesday at a event called the Pro-Human Assembly in Washington, DC, Bernie Sanders and Steve Bannon shared a stage where they denounced tech oligarchs and called for reining in AI before it exceeds human control. Look, they've both been saying these things separately for a while. That's not new information. The fact that both of them thought that it was a good idea and high time for them to get on stage together fascinated me. Again, we already knew how they felt about this. To do it together, fascinating. Bannon's proposals very much differ from Sanders. He doesn't trust Congress. He wants Trump to slow development, executive order, et cetera, et cetera. Sanders is a man of the people, really wants some congressional oversight here. But there were some really interesting things that came up in this conference that I'm still thinking about, again, on stage with both of these people. The conference included parents whose teenagers had died by suicide after chatbot relationships, talks about child safety regulation, AI-generated nonconsensual sexual imagery. There was a lot of stuff which would be totally normal and not out of place on one of these stages, but again, it was all of these people together. And Trump meanwhile is just continuing to slow down all of this. He posted that the "only controls or guardrails that AI needs is a capitalized strong and smart (high IQ) president, and the USA has that in spades." So that's where we're at, folks.
  Zoë Schiffer: I have two things to say. Number one, Steve Bannon, the only thing I want to hear from you is the unreleased audio from your All-In episode that never occurred. Please, God, someone send it to me. Number two, have any of these people articulated why China winning the AI race is so ... Are they worried that China's hacking abilities get so good and scary or that Chinese AI proliferates and influences America and everyone turns communist? What's the scenario?
  Brian Barrett: I think it's more simple than that, maybe. I think it's economic. I think that so much of the US economy is propped up on AI right now that if China becomes the world leader in AI and our industry crumbles because of that, we're in trouble. I think there are also security concerns, but I think fundamentally, if the US is not the leader in AI, everything kind of crumbles economically.
  Zoë Schiffer: Yeah, no, I get that. And again, we have talked about Chinese open models being very close to as good as some of the models from the leading US frontier labs, and obviously a lot cheaper if they're open source. I completely get that. I think I'm craving so much specificity in this conversation because I just feel like Dario, Sam, all of the politicians, they're so vague in how they talk about this and they've been doing it for so long, the threat of China in all of these different ways, the threat of RSI, AGI, that I'm like, "What actually are we talking about?"
  Brian Barrett: I have one more thing on Trump. We're going to have a story from Maddy Varner, a WIRED senior writer about the dynamics here. But basically the reason that these AI labs are saying we're not sure if we can slow down, we're not sure if we can pace the frontier, is that we're worried that it will be an antitrust violation. We are worried that the FTC is going to come after us with an enforcement action because we are colluding, et cetera.
  Leah Feiger: Now you're worried about an antitrust violation?
  Brian Barrett: Yeah, but that's what they're saying. And Donald Trump has effectively turned every enforcement branch of the US government into his own personal attack dogs. So if Donald Trump is saying, "You need to keep building AI as fast as you can," and if these AI companies collectively say, "Actually, we're going to slow down," regardless of the merits of it, you could pretty easily see an antitrust case popping up real quick as a way to say, "No, you're not slowing down at all." So by making these overt threats on Truth Social, I think he is really making it very hard for these AI companies to do any kind of meaningful slowdown, any kind of pullback.
  Leah Feiger: Coming up after the break, we'll share our WIRED TIRED picks for the week. Stay with us.
  Zoë Schiffer: OK, it's time for our WIRED TIRED segment. As usual, whatever is new and cool is WIRED, whatever passé thing we're totally over is TIRED. Leah, do you want to go first?
  Leah Feiger: OK. My TIRED is, while I often get really lovely comments on social media and emails from our avid readers and listeners about how much they love Wired.com and love Uncanny Valley, I also sometimes get emails from people who are asking me for specific information on logins and reading articles and accessing different WIRED bits and bobs. So that's my TIRED, getting those ... I'm sorry, if you message me and ask for help getting anything WIRED, Condé Nast, I have nothing. I don't know how to do it. I barely know how to use my own phone. So that's my TIRED.
  Brian Barrett: Leave Leah alone. OK.
  Leah Feiger: Leave me alone. My WIRED is we have a new app, and it is so beautiful. Just launched this week. Highly recommend checking it out, downloading it. As someone who really doesn't like adding additional little pieces, additional layers to my various routines, it's fantastic. It's really, really fantastic. It's so easy to use, connects so simply to subscriber everything. Highly recommend. I already love it. I have a feeling that this is going to change your life as well as mine, and no one will email me again.
  Brian Barrett: It's a high part—
  Zoë Schiffer: And Brian, what do you have?
  Brian Barrett: My TIRED is, OK, it's not a new thing, but we talked about it today. AI lingo, AI industry lingo. I'm over it. I'm sorry. AI labs instead of companies. It's the frontier instead of just good tech. We're pacing it. We're pacing the frontier. What is that? Get it together. I know this has been part of the tech industry forever, right?
  Leah Feiger: No, drag them. This is people who read too much Lord of the Rings and watched too much Star Trek who've tried to integrate cultural touchstones of their childhood into our day-to-day lives, and I hate it.
  Brian Barrett: It's also a way to make it feel inaccessible, right? It's like, "Oh, they need to pace the frontier. Well, I don't know anything about that, but it sounds important. Oh, AI has an alignment problem. Oh, that sounds serious." It's really like AI's misbehaving. AI is not doing what you want it to. You don't have control over your stuff. So I think it's a way to sort of obfuscate the issues, and that's becoming increasingly clear, especially as those issues have become more severe, which we've spent a lot of time talking about. So I am tired of it. That's my TIRED. My WIRED, Leah, you stole mine. I won't spend much time on it. The WIRED app, it is, it's great. A lot of work went into it. I have made my family members download it. They like it, or at least they tell me they do. So please go ahead and check it out and leave a review. It really helps.
  Zoë Schiffer: That's our show for today. We'll link to all the stories we spoke about in the show notes. Adriana Tapia produced this episode. It was mixed by Pran Bandi, who's also our New York studio engineer. It was fact-checked by Matt Giles and Daniel Roman. Kate Osborn is our executive producer, and Katie Drummond is WIRED's global editorial director.
  Comments
  Back to top