内容来源：https://www.wired.com/story/ai-agents-hacking-systems-could-push-the-us-and-china-to-cooperate/

内容总结：

中美AI安全合作：从零和博弈到共同求存？

长期以来，人工智能领域的竞赛被视为一场零和博弈：最终要么美国胜出，要么中国胜出。然而，随着AI模型——尤其是AI智能体——能力日益增强，由此引发的担忧也随之加剧，两国研究人员正试图携手合作，共同应对AI安全问题。

《连线》杂志特约编辑佐伊·希弗本周与资深撰稿人威尔·奈特展开对谈。奈特分享了今年夏天访问中国时的所见所闻，并探讨了为什么中美两国或许真的需要开始合作，以避免一场重大的AI灾难。

中国AI专家也“慌了”

奈特指出，过去一年里，他注意到来自中国的AI安全研究显著增多。在北京参加的一场由当地实验室举办的会议上，AI安全成为核心主题。无论是实验室还是公司，AI安全问题被频繁提及。值得注意的是，中国研究人员与美国同行担心的事情如出一辙：黑客滥用AI系统，或是系统本身失控。

中国更务实，美国更逐梦

与硅谷对“通用人工智能（AGI）”的狂热追求不同，中国的AI发展路径显得更为务实。人们更关心AI如何在实际中发挥作用，对“创造数字之神”这类想法兴趣寥寥。这种实用主义导向，使得中国在快速采用AI智能体等新技术时，也格外关注其可靠性和安全性，聚焦于如何防止出错。相比之下，美国若一味追求“智能奇点”，则可能更侧重不惜一切代价的进步。

开源与“蒸馏”之争

针对美国业界批评中国公司通过“蒸馏”技术抄袭美国前沿模型的指控，奈特认为这种说法过于简单化。AI发展本身就是全球协作的产物，美国公司同样会借鉴其他公司的成果。他特别指出，像DeepSeek、月之暗面（Kimi）等中国模型，本身就包含了许多独特的工程创新，并非简单的复制。将中国描绘成单纯的“抄袭者”是危险的，这会让美国产生不切实际的优越感，而中国公司未来无疑会展现出更强的自主创新能力。

“切尔诺贝利时刻”的阴影

奈特提到，MIT教授斯蒂芬·卡斯珀在会上表示：“目前几乎所有AI研究者都能达成共识的一点是，AI不需要一个切尔诺贝利时刻。” 这种担忧并非指AI统治人类，而是指随着AI系统在金融、网络安全等关键领域被广泛部署，其高度复杂和不可预测性可能导致突发的、破坏性的重大事件，例如由AI引发的金融闪电崩盘，或是AI智能体发起的大规模网络攻击，从而酿成国际危机。这正是两国研究者共同的关切。

合作的可能性与障碍

尽管地缘政治紧张，但奈特观察到，中国方面并不认为必须击败美国才算成功，竞争并非纯粹的零和游戏。不少中国研究人员渴望与美国同行合作，特别是在AI智能体安全等前沿领域，但美国的出口管制等限制措施，为这种科研合作设置了巨大障碍。

硬件领域的“非零和”启示

奈特以英伟达最近发布的采用中国宇树科技机器人身体、搭配英伟达美国芯片的人形机器人为例，指出这体现了中美科技合作的可能性。事实上，美国几乎所有机器人研究实验室都在使用中国制造的、物美价廉的机器人平台。美国试图完全与中国制造业脱钩，不仅不现实，可能还会损害自身利益。

归根结底，奈特认为，美国不应只想着如何遏制中国，更应思考如何通过加大对基础科学和技术的投资来保持自身的竞争优势。在AI安全等关乎全人类福祉的议题上，中美两国的合作不仅是必要的，而且迫在眉睫。

中文翻译：

人工智能竞赛长期以来一直被描述为零和博弈：最终要么美国赢，要么中国赢。但随着人们对人工智能模型——尤其是人工智能代理——能力不断增强的担忧日益加剧，两国研究人员正试图联手开展人工智能安全研究。本周，特约编辑佐伊·希弗与资深撰稿人威尔·奈特聊了聊他今夏访问中国时的所见所闻，以及为什么两国可能真的需要开始合作以避免一场重大人工智能灾难。

这是我们暑期系列节目的第二集。下周我们将带着常规圆桌讨论回归。

本集提到的文章：

• 《我会见了中国顶尖人工智能专家。他们也非常担忧》
• 《人工智能黑客攻击很糟糕。人工智能蠕虫和病毒会更糟》
• 《未来的人形机器人是一个六英尺高的壮汉，有着中国的身体和美国的大脑》

你可以在Bluesky上关注佐伊·希弗，账号为@zoeschiffer，以及威尔·奈特，账号为@willknight。写信给我们：[email protected]。

如何收听
你总是可以通过本页上的音频播放器收听本周的播客，但如果你想免费订阅以获取每一集，方法如下：

如果你使用的是iPhone或iPad，打开名为“播客”的应用，或者直接点击此链接。你也可以下载Overcast或Pocket Casts等应用，搜索“uncanny valley”。我们在Spotify上也有。

文字记录
注意：这是自动生成的文字记录，可能包含错误。

佐伊·希弗：这里是《WIRED》的《Uncanny Valley》。我是特约编辑佐伊·希弗。如果你这个夏天一直在关注科技新闻，当然如果你一直在收听本节目，你可能已经知道中国频繁登上头条，尤其是在人工智能竞赛方面。该国的开放模型继续缩小与美国前沿模型的差距，据一些人估计，其成本仅为美国的零头。反过来，美国维持着对芯片和出口管制的严格限制，以减缓中国的崛起。我们在很多方面将人工智能进步视为零和博弈——如果中国赢了，美国就输了，反之亦然。但有一个担忧真正跨越了这条界线，那就是人工智能安全。

档案音频：在OpenAI和Anthropic的人工智能代理多次突破其隔离环境的实例之后，人工智能网络安全风险一直备受关注。

佐伊·希弗：人工智能代理黑客攻击平台的消息在夏天加剧了这个问题的紧迫性。相应地，政府官员被迫关注人工智能监管并采取行动。

档案音频：特朗普总统签署了一项行政命令，要求科技公司在新人工智能模型公开发布之前让政府进行监督。

佐伊·希弗：那么，美国和中国真的能受益于合作吗？要实现这一点又需要什么？今年夏初，《WIRED》的高级记者威尔·奈特访问了中国，去寻找一些答案。威尔·奈特，非常感谢你来到这里。

威尔·奈特：谢谢你的邀请。

佐伊·希弗：好的，我想从你的旅行开始聊。你今年夏初去了中国，当时我们在美国听到关于人工智能安全的消息还不多。事实上，感觉随着特朗普在白宫的第二个任期，一切都变成了“美国必须赢”的框架，人工智能安全甚至开始听起来像是反增长的。但我很好奇，你在人工智能安全方面在中国听到了什么、看到了什么。

威尔·奈特：是的，回溯到大约一年或半年前，我注意到来自中国的人工智能安全研究明显增多了。所以，我去了北京参加一个会议，这个会议由当地的一个实验室主办。他们在北京、上海和其他地方都有这样的实验室。事实证明，人工智能安全是一个非常重大的主题。很明显，这是研究人员感兴趣的事情。而且实际上，仅仅是参观实验室和公司时，人工智能安全的问题也经常被提到。

佐伊·希弗：我能问一下吗，当他们谈论人工智能安全时，这等同于防护措施吗？因为我们也知道中国确实全力押注开放模型，而开放模型的意义就在于人们可以下载、修改它们，并出于任何目的使用它们。

威尔·奈特：嗯，事情没那么简单，因为在例如中国，你的模型能说什么受到更多控制，而且实际上围绕人工智能有相当多的法规。所以公司构建这些开放模型，但任何将它们放到互联网上的人都必须对自己的行为非常小心。而最近，人们对代理和OpenClaw这类东西产生了巨大兴趣。这是一个非常大的主题。对我来说有趣的一点是，至少在对比中国和美国的人工智能时，那里的人们似乎不那么迷恋AGI（通用人工智能）和创造这个“数字神”的想法，而是更关心这实际上怎么才能有用，无论你是商人还是个人，是否真的会去使用它。所以，很多人对OpenClaw这类东西产生了浓厚兴趣，而且在中国这常常是常态——人们迅速采用，然后看到它可能出什么问题。所以有很多关注点在于：我们如何让这些东西变得可靠？

佐伊·希弗：是的，有道理。我的意思是，当你说中国更专注于经济上有用的模型时，这似乎确实需要一个包含一定程度的稳定性、可靠性、防护措施和安全性的框架。而如果你专注于达到神一般的智能，即AGI，那么也许你更专注于不惜一切代价的进步。

威尔·奈特：对，我认为是这样。我去的那个会议，主题之一是“代理安全”。鉴于我们现在看到所有这些人工智能代理黑客攻击的问题，网络安全在那里是一个非常重要的议题。看起来人们担心的事情和美国的人完全一样。他们担心黑客滥用这些东西，或者担心这些系统失控。正如你提到的，我认为现在有一种感觉，而且这种感觉更强烈了，那就是美国和中国很可能需要在某些问题上合作，以避免不可预测的系统性问题，同时也为了围绕这些系统制定更多行为规则。

佐伊·希弗：但这实际上会是什么样子呢？会是一套美中共同达成的协议吗，几乎就像美国研究人员最近呼吁美国政府在人工智能发展速度上设定步伐那样？会是那样的东西，还是更技术性的东西？

威尔·奈特：我认为这是很多研究人员希望或呼吁的事情，就是某种形式的协议。我不知道在华盛顿和北京之间，他们的谈判一直非常艰难——很难预测那些谈判会如何发展。但仅仅是围绕沟通的某些规则。在军事局势等情况下，是有沟通渠道的。所以如果出了什么错，如果有人工智能系统开始做非常激进的事情或攻击系统，你有一种方式可以说：“这是个错误。”所以，类似的东西也可能在考虑之中。但我认为这同样是一个双方如何建立信任的问题，因为实际上，尤其是在网络安全方面，很长一段时间以来几乎没有合作，因为一直存在双方互相黑客攻击、无法就规则达成一致的情况。所以，上周我去拜访了一位网络安全和人工智能研究员，他正在做一些非常棒的工作，我将在下一期《AI Lab》通讯中报道。他说他无法与美国研究人员合作，因为他们不被允许，因为存在某些限制。他最近开发了一个基准测试，用于测试人工智能模型的黑客攻击能力，他想让美国公司参与，但他们不太确定该怎么做。所以，我认为看到更多的合作，哪怕是公司之间的合作，都是一件好事。我们看到美国公司对中国公司非常挑剔，但实际上，每个人都有充分的理由共同努力，确保事情不会出错。

佐伊·希弗：嗯，我想深入谈谈这个，但我也想说，这种合作的想法，乍一听非常学术，甚至有点天真。这是一个好主意，但这实际上会怎么运作呢？因为我的看法是——我就直说了——这个想法是我与许多在美国从事前沿人工智能工作的公司交谈后得出的，但他们让我相信，确实存在相当程度的“蒸馏”现象。也就是说，中国在蒸馏美国的前沿人工智能模型，这造成了一种局面：他们能够在借鉴美国创新的基础上，拥有非常强大的开源模型，而且这些模型更便宜、更高效。但我很好奇你怎么看这个，然后你在中国交谈过的研究人员对这个指责怎么看。

威尔·奈特：是的，我认为，这是个很好的观点，也是一个非常重要的主题。我们听到人们批评中国公司进行蒸馏。这种做法的原理是，通过获取另一个模型的输出结果来训练你自己的模型，这是一种获取其中大量内嵌知识的捷径。但事实是，人工智能是由来自世界各地的研究人员在不同的公司和实验室共同构建的。有很多人原本来自中国，也许是在美国受的教育，在美国公司工作。而且，由于这些人参加同一会议、彼此认识，而且这是开放科学，有大量的工作被共享，这对进步非常有益，而平衡这一点就是挑战之一。确实，中国公司蒸馏了美国的模型，但美国公司也蒸馏过其他美国公司的模型。这通常是启动新模型工作的一种方式，而且在学术界非常普遍，很多研究人员都会这样做。我要说一点，我觉得有点讽刺的是，这些通过抓取海量受版权保护的内容来建立自己商业模式的公司，现在却在抱怨他们的模型被复制，或者——

佐伊·希弗：嗯，我认为这就是为什么他们必须大肆谈论中国这样做，因为如果他们谈论美国有人这样做，立即会遭到批评：“得了吧，你们拿走了所有的书，你们未经许可拿走了一切。”但当它被描述为中国窃取美国时，味道就有点不同了。

威尔·奈特：是的。这符合一种有一定合理性的叙事，即中国公司在抄袭，但我认为这过于简单化和片面了。你可以看看深度求索的模型。他们确实做了非常重要、独特的创新，美国公司也抄袭了这些创新。最近被指责蒸馏的中国最新模型，是月之暗面的Kimi。他们发布的研究论文中包含了许多非常、非常有趣的创新，工程上的创新。所以，说中国仅仅是抄袭，这确实不是事实。我认为，美国政府和企业认为他们拥有这种天赐的优势是危险的，因为我们很可能看到中国公司会越来越有创新能力，也会走自己的路。

佐伊·希弗：广告之后我们马上回来。请继续收听。我很好奇，当我们谈论安全时，我感觉在美国，当然在特朗普的第二个任期内，安全已经开始让人觉得，至少从政府的角度来看，它在某些方面是反增长的。我认为随着最近关于OpenAI和Anthropic的模型黑客攻击其他公司的报道，这种情况正在略微改变。但在此之前，感觉我们确实不太想谈论太多人工智能安全。中国是否将人工智能安全等同于反增长，还是他们对此有不同的看法？

威尔·奈特：我认为他们有根本不同的看法。这只是我的个人观点，但我认为美国政府的那个叙事，即安全在某种程度上是“觉醒的”、意味着太多监管，这归根结底不过是试图让模型真正工作而已。事实是，让模型可靠与让它成功是完全兼容的。我认为中国这边的看法更像是：我们希望这些代理不行为不端，那么它就会更成功、更有价值。所以，上周我参观了上海复旦大学的一个人工智能计算机科学实验室，那里的一位教授正在研究人工智能代理不仅可能做出不可预测的事情，比如像我们听说的OpenAI和Anthropic的代理那样黑客攻击其他系统，而且还会主动寻找方法来复制自己到其他系统，寻找资源并适应以逃脱控制。他的工作表明，只要稍加推动，模型就会这样做。所以，这就像一种计算机蠕虫，它不仅仅是稍微修改自身以逃避控制，而是真正地在网络中侦查，找出如何入侵下一个系统，也许发现软件漏洞，然后把自己复制到别处。我们完全有可能看到未来的人工智能代理做这种事。

佐伊·希弗：好吧。你说话的时候我的心都在怦怦跳。我的意思是，第一步就是理解，它们怎么会这样做？我真诚地希望第二步是，我们如何阻止它们这样做？

威尔·奈特：是的。所以不，绝对不是。他这样做绝对是为了试图理解如何预防。而且他非常想做的事情之一就是与美国研究人员合作。他说这是我们都应该意识到的一件非常重要的事。

佐伊·希弗：我的意思是，尽管特朗普的言辞在某些方面非常反华，但斯科特·贝森特已经与他的中国同行取得了进展。我感觉在某种程度上，我们还是认识到需要合作的。我很好奇，中国如何看待与美国的竞赛？因为我们确实几乎把中国看作一个“狼来了”的角色，在某种程度上是激励人们更努力、更快工作的力量。但在那边是类似的情况，还是非常不同？

威尔·奈特：是的，我认为在中国，我得到的印象是，人们确实感到在与美国竞争，并且面对特朗普的压力确实很大，但竞争没那么像零和博弈。你不必打败美国才能成功，反之亦然。

佐伊·希弗：所以，斯蒂芬·卡斯帕，麻省理工学院的著名计算机科学家，也参加了你参加的那个会议，他告诉你：“现在几乎所有人工智能领域的人都能达成一致的一件事是，它不需要一个‘切尔诺贝利时刻’。”你能谈谈这意味着什么，以及为什么这对斯蒂芬和你交谈过的其他人来说如此重要吗？

威尔·奈特：是的，我认为随着我们看到这些模型变得越来越强大，拥有更多的自主性，并且被更广泛地部署，出现不可预测的大问题的途径就越来越多了。一个例子——实际上是我和一些中国研究人员交谈时提到的——是人工智能在金融和交易中的使用。对吧？你将会有越来越不透明、更强大、更高速的系统，它们更加不可预测。我认为美国和中国都会非常希望避免某种由人工智能系统不可预测的行为驱动的金融闪崩或崩溃。对我来说，这比人工智能接管世界的想法更令人担忧，或者也有人担心它可能被武器化，被恐怖组织之类的利用。但我只是认为，随着这些系统变得更强大、更具代理性，它们自我改进、以我们无法总是预测的方式变得更强大的想法，那里的研究人员和这里的研究人员有着同样的担忧。所以在我看来，我认为“切尔诺贝利”事件会是某种金融闪崩，或者是一个人工智能代理进行疯狂黑客攻击，引发重大国际事件，诸如此类的事情。

佐伊·希弗：我想快速转换一下话题，谈谈硬件，因为NVIDIA最近公布了一个人形机器人的蓝图，将宇树科技的中国制造身体与NVIDIA的美国芯片配对。鉴于人工智能竞赛在美国和中国之间变得多么具有政治敏感性，你对这样的合作关系有什么反应？

威尔·奈特：我认为这是黄仁勋和NVIDIA推动美中之间建立更友好关系的一部分，如果他们想销售大量芯片，这可能会对他们有利。但我认为在某种程度上，这也代表了这种不那么零和的理念。所以我认为这是一种声明，试图说：“看，你们可以合作。”美国已经表示将禁止来自中国的新人形机器人。这里需要注意的是，几乎所有美国机器人研究实验室都使用宇树科技的小型人形机器人，因为它真的很便宜。而美国如果不在工业政策上投入大量精力来建立自己的制造能力，就无法与中国的制造业竞争，而这将需要几十年的时间。所以事实是，有一种方式可以合作并在当下获益。问题是，这从长远来看是否会损害美国的利益？美国是否必须拥有自己的机器人？我认为，有机会尝试建立美国自己的机器人产业以及其他许多产业。我认为这比禁止中国的机器人更重要。

佐伊·希弗：我也觉得这印证了我们对出口管制的一些批评。我们听到有人说：“哦，不，让中国依赖美国技术更好。如果我们切断供应，他们就会开发自己的硬件，”这确实感觉正在发生。我们听到过其他版本的这个说法，比如深度求索推出他们的前沿模型时，那些模型看起来非常、非常强大，然后就是：“哦，不，我们因为不给他们那么多芯片，反而促使他们变得高效得多。”

威尔·奈特：是的。我六月份在中国的时候，确实看到了华为的新人工智能硬件。这家公司长期受到美国制裁，一直在开发一个旨在取代或与NVIDIA硬件竞争、用于训练人工智能模型的系统。他们做了一件非常聪明的事：他们采用了性能较低的芯片，并利用他们在光纤网络方面的专业知识，将大量芯片组合在一起。这消耗更多电力，所以效率较低，但它可以达到接近NVIDIA的水平。它没有那么好，但现在很多人在使用它，因为他们不确定NVIDIA是否是一个可靠的供应来源。事实是，它们不如NVIDIA强大，所以目前确实给了美国优势。但在我看来，我认为最重要的事情之一是，美国必须超越仅仅考虑“我们如何掐住中国”的思路，而要问：“我们如何假设他们会变得更好、更有竞争力，然后超越他们？”对吧？这在每个行业都是如此，比如制造业和机器人技术。你必须问：“我们如何投资于基础科学和技术的进步？”中国正在这样做。与此同时，美国却在削减科学资金，我个人认为这简直是最大的丑闻。

佐伊·希弗：威尔·奈特，非常感谢你来到这里。

威尔·奈特：谢谢你的邀请。我想用中国人们通常道别的方式说再见，那就是“拜拜”。

佐伊·希弗：这就是我们今天的节目。我们会在节目说明中链接所有讨论过的文章。阿德里安娜·塔皮亚制作了本集节目。混音由普兰·班迪完成，他也是我们纽约工作室的工程师。事实核查由马特·吉尔斯和丹尼尔·罗曼负责。凯特·奥斯本是我们的执行制片人，凯蒂·德拉蒙德是《WIRED》的全球编辑总监。

评论
返回顶部

英文来源：

The AI race has long been framed as a zero-sum game: Either the US or China will win in the end. But as concerns pile up around the increasing capabilities of AI models—especially AI agents—researchers in both countries are trying to team up to work on AI safety. This week, contributing editor Zoë Schiffer speaks with senior writer Will Knight about what he saw and heard on the ground when he visited China this summer—and why the two countries might actually need to start working together to avoid a major AI catastrophe.
This is our second episode from our summer-break series. We’ll be back next week with our usual roundtable.
Articles mentioned in this episode:

• I Met With China’s Top AI Experts. They’re Freaking Out, Too
• AI Hacks Are Bad. AI Worms and Viruses Will Be Worse
• The Humanoid Robot of the Future Is a 6-Foot-Tall Beefcake With a Chinese Body and an American Brain
  You can follow Zoë Schiffer on Bluesky at @zoeschiffer and Will Knight on Bluesky at @willknight. Write to us at [email protected].
  How to Listen
  You can always listen to this week's podcast through the audio player on this page, but if you want to subscribe for free to get every episode, here's how:
  If you're on an iPhone or iPad, open the app called Podcasts, or just tap this link. You can also download an app like Overcast or Pocket Casts and search for “uncanny valley.” We’re on Spotify too.
  Transcript
  Note: This is an automated transcript, which may contain errors.
  Zoë Schiffer: This is WIRED's Uncanny Valley. I'm Zoë Schiffer, contributing editor. If you've been following tech news this summer, and definitely if you've been listening to the show, you probably already know that China has been in the headlines quite a lot, particularly when it comes to the AI race. The country's open models continue to close the gap with US frontier models at what some people estimate is a fraction of the cost. In turn, the US has maintained its tight restrictions on chips and export controls to slow down China's rise. We think of AI advancement in so many ways as zero-sum, if China wins, the US loses, and vice versa. But there's a concern that really stretches across those lines, and it's about AI safety.
  Archival audio: AI cybersecurity risks have been top of mind after multiple instances of AI agents from both OpenAI and Anthropic breaking out of their enclosures.
  Zoë Schiffer: News of AI agents hacking platforms added urgency to this issue over the summer. In turn, government officials have been forced to pay attention and act on AI regulation.
  Archival audio: President Trump has signed an executive order asking tech companies to give the government oversight of new AI models before their public release.
  Zoë Schiffer: So, could the US and China actually benefit from working together? And what would it take to even make that happen? Earlier this summer, WIRED's senior correspondent Will Knight visited China to get some answers. Will Knight, thank you so much for being here.
  Will Knight: Thanks for having me.
  Zoë Schiffer: OK, so I want to start with your trip. You went to China earlier this summer, and at the time, we weren't hearing as much about AI safety in the US. In fact, it felt like with Trump's second term in the White House, it was like a the-US-needs-to-win framing, and AI safety almost started to sound like anti-growth. But I'm curious what you were hearing and seeing in China on the AI safety front.
  Will Knight: Yeah, so going back maybe a year or six months, I'd noticed a lot more AI safety research coming out of China. And so, I went to this conference in Beijing, put on by one of the city-located labs that they have there. They have these ones in Beijing and Shanghai and elsewhere. And it turns out that AI safety was a really big theme. It's very clear that it's something that researchers are interested in. And actually, also just visiting labs and companies, the question of AI safety came up a lot.
  Zoë Schiffer: Can I just ask, when they're talking about AI safety, does it translate to guardrails? Because we also know that China has really gone all in on open models, which I mean, the whole thing is that people can download and tweak them and use them for whatever purposes they want.
  Will Knight: Well, it's not totally that simple because in China, for example, what your models can say is more controlled, and there are actually quite a lot of regulations around AI. So companies build these open models, but then anybody putting them on the internet has to be very careful about what they do. And then more recently, there's been a huge interest in agents and things like OpenClaw. That's been a really big theme. And one of the things that's interesting to me, at least when it comes to contrasting AI in China and the US, is that people there seem less enamored with the idea of AGI and creating this digital god, are more like, how is this actually going to be useful and whether it's you as a business person or an individual actually using it. So, a lot of people got very interested in, and it's often the case in China, very rapidly adopted things like OpenClaw and then saw how it could go wrong. So there's a lot of focus on How do we make these things reliable?
  Zoë Schiffer: Yeah, it makes sense. I mean, when you're talking about China being more focused on economically useful models, that seems like a framework that does require a certain amount of stability, reliability, guardrails, safety. Whereas if you're focused on reaching godlike intelligence, i.e. AGI, then maybe you're more focused on just advancement at all costs.
  Will Knight: Right, I think that's right. The conference I went to, one of the themes was agentic safety. Given that we're now seeing all these issues with AI agents hacking things, cybersecurity was a really major topic there. It seems people were worried about exactly the same thing as folks in the US. They're worried about hackers misusing these things or about these systems running amok. And as you alluded to, I think there's a sort of sense now, a little more of a sense that the US and China might well need to work together on some of these things to avoid unpredictable systemic issues, as well as just to set more rules of the road around these systems.
  Zoë Schiffer: What would that actually look like, though? Would it be like a set of agreements that the US and China make together, almost like what US researchers were calling for recently, in terms of the US government setting the pace of AI development? Would it be something like that or something more technical?
  Will Knight: I think that's something that I think a lot of researchers are hoping for or calling for, is something like that where there's some sort of agreement. I don't know when it comes to Washington and Beijing, their negotiations have been very hard for—and it's really difficult to predict how those would shake out. But just some sorts of rules around communication. And in cases like military situations, there are lines of communication. So if something happens that goes wrong, if you have an AI system that starts doing something very aggressive or attacking systems, you have a way to say, "This is a mistake." So, something like that might also be in the offering. But I think it's also a question of how the two sides build trust as well, because actually, especially when it comes to cybersecurity, for a long time there's been not very much cooperation at all, because it's been a case of either side hacking each other and failing to agree, the rules of the right. So actually, last week I went to visit a cybersecurity and AI researcher who was doing some really fantastic work that I'm going to write about for my next AI Lab newsletter. And he was saying he can't collaborate with US researchers because they're not allowed to because there are certain kind of restrictions. He'd recently developed this benchmark to test the cybersecurity, the hacking capabilities of AI models, and he wanted to get US companies to participate but they weren't really sure how to do that. So, I think it would be a good thing to see a lot more collaboration even between the companies. We see the US companies being very critical of Chinese ones, but actually, there's a lot of good reason for those for everybody to work together to make sure things don't go wrong.
  Zoë Schiffer: Well, I want to get into that, but I also wanted to say that this idea of collaboration, when you first say it, it sounds very academic, almost naive. It's a nice idea, but how would that actually work? Because my perception is, and I'll just be upfront, that I get this idea from talking to a lot of companies that work on frontier AI in the US, but they have convinced me that there was a fair amount of distillation that went on. That China was distilling frontier AI models, and that that has created a situation where they are able to have very capable open-source models that are a lot cheaper and more efficient, built on the back of US innovation. But I'm curious what you think about that, and then what researchers you spoke to in China think about that accusation.
  Will Knight: Yeah, I think, I mean, that's a great point and that's a really important theme. We hear people criticizing Chinese companies for distilling, for doing this distillation. So you teach your model by taking the output of another model, and that is a shortcut to learning a lot of the stuff that's embedded. But the truth is that AI has been built by researchers from all over the world working at different companies and different labs. There are many, many people who are originally from China, maybe educated in the US, working at US firms. And also because these people go to conferences and know each other and this is open science, there's an enormous amount of work that is shared and then that is very beneficial to progress, and balancing that is one of the challenges. It's true that Chinese companies have distilled US models, but so have US companies done that to other US companies. It generally is a way that you get a kickstart working on a new model, and it's very widely done in academia, actually. A lot of researchers will do that. I would say one thing, I find it a little ironic that these companies that have built their businesses by scraping enormous amounts of copyrighted content are now complaining about their models being copied, or—
  Zoë Schiffer: Well, I think that's why they have to talk about China doing it so much, because if they talk about anyone in the US doing it, the immediate criticism is, "Well, come on, you took all of the books, you took everything without permission." But when you'd frame it as China stealing from the US, it has a slightly different flavor.
  Will Knight: Yeah. It fits a narrative that has some legitimacy of Chinese companies copying, but I think it is much too simplistic and limited. And so, you can look at things like DeepSeek's model. They did really, really important innovation, unique innovation that other, the US companies have copied. The latest model from China, which has been accused of this distillation, Kimi from Moonshot. The research paper that they put out includes a lot of really, really interesting innovations, engineering innovation. So, it's really not the case that China is simply copying. And I think it's dangerous for the US government and companies to believe that they have this sort of God-given advantage, because I think we are going to see probably Chinese companies being more and more innovative doing their own thing as well.
  Zoë Schiffer: We'll be right back after the break. Stay with us. I'm curious, when we talk about safety, I feel like in the US, certainly in Trump's second term, again, safety has started to feel like, at least from the administration's perspective, it is anti-growth in certain ways. I do think this is changing slightly with the recent stories regarding OpenAI and Anthropic's models hacking into other companies. But prior to this, it felt like we were really not wanting to talk so much about AI safety. Does China equate AI safety as anti-growth, or do they have a different perspective on that, overall?
  Will Knight: I think they have a fundamentally different perspective. This is just my opinion, but I think that narrative, that view from the US government was that it was somehow woke and too much regulations, and it goes to just trying to make the models really work. The truth is that making a model reliable is entirely compatible with making it successful. I think that's more the view from China, is like, we want these agents not to misbehave, then it will be more successful and higher value. So last week, I visited this computer science lab in Fudan University in Shanghai, where a professor's working on exploring how AI agents could not just do unpredictable things like hack other systems as we've heard about OpenAI's and Anthropic's agents doing, but actually look for ways to replicate, to copy themselves over to other systems, to seek out resources and to be adaptive to escape control. And his work shows that the models will do that with a little bit of nudging. So, it'd be like a computer worm that doesn't just modify itself slightly to evade control, but actually looks around a network, figures out how to hack the next system, maybe finds software vulnerabilities, copies itself somewhere else. It is entirely possible that we'll see future AI agents do that.
  Zoë Schiffer: OK. My heart's beating as you're talking. I mean, step one is just understanding, how would they do this? I sincerely hope step two is, how do we stop them from doing this?
  Will Knight: Yeah. So no, absolutely. He's absolutely doing this as a way to try and understand how to prevent it. And one of the things he really wants to do is work with US researchers. He says this is a really important thing we should all be aware of.
  Zoë Schiffer: I mean, it does seem like, although the rhetoric from Trump has been very anti-China in certain ways, Scott Bessent has made inroads with his counterpart in China. I think there's been some level of, we do need to work together. I'm curious, how does China think about the race against the US? Because we really see China almost as a boogeyman and it's like this force that's galvanizing people to work harder and faster in some ways. But is it similar over there or very different?
  Will Knight: Yeah, I think in China, the impression I get is that what people feel that they're in competition with the US and facing certainly a lot of pressure with Trump, that it's less of a zero-sum game. That you don't have to beat the US to be successful, and vice versa.
  Zoë Schiffer: So Stephen Casper, a renowned computer scientist at MIT who spoke at the conference that you attended, told you that, "One thing that almost everyone in AI can agree on right now is that it doesn't need a Chernobyl moment." Can you talk about what does that mean and why did it feel so important to Stephen and other people you spoke to?
  Will Knight: Yeah, I think as we're seeing these models get more capable and have more agency and being deployed more widely, there is just lots more ways in which you might have unpredictable and big problems. One example which actually came up with some Chinese researchers that I spoke to was use of AI in finance and in trading. Right? You'll have increasingly opaque, more capable, high speed systems that are more unpredictable. And I think both the US and China would be very keen to avoid some sort of financial flash-crash meltdowns that would be driven by AI systems just behaving unpredictably. To me, I think that's more the concern than the idea of AI taking over, or there is also worry that it could be weaponized and used by terrorist groups or something like that. But I just think as these systems get much more capable, more agentic, the idea of them improving themselves, getting more capable in a way that we can't always predict, researchers there and here just have the same concern. And so to my mind, I think that the Chernobyl incident would be some sort of either a financial flash-crash or an AI agent that went on a hacking spree that causes major international incident, that sort of thing.
  Zoë Schiffer: I want to shift gears really quickly to talk about hardware, because NVIDIA recently unveiled a blueprint for a humanoid robot that pairs Unitree's Chinese-made body with NVIDIA's American chips. Given how politically loaded the AI race has become with US and China, what was your reaction to a partnership like this?
  Will Knight: I saw that as part of Jensen and NVIDIA's push to try and have a more friendly relationship between the US and China, which probably serves them well if they're trying to sell a lot of chips. But I think it's also representative of this less zero-sum idea, to some degree. And so I saw it as a kind of a statement to try and say, "Look, you can work together." The US has said it's going to ban new humanoids from China. And the thing to note here is that pretty much all US robotics research labs use Unitree, this small humanoid because it's really cheap. And the US just can't compete with China's manufacturing without spending a lot of effort, industrial policy to build up its own, which would take decades. And so the truth is, there is a way that you can collaborate and have benefits now. The question is, does that undermine US interests longer term? Does the US have to have its own robots? I would think that there's some opportunity to try and build up the US's own robotics industry, as well as many of its other industries. That is more important, I think, than banning China's.
  Zoë Schiffer: I also feel like it speaks to some of the criticisms of export controls that we heard. We heard people being like, "Oh no, it's better for China to be dependent on US technology. If we cut them off, they're going to develop their own hardware," which does feel like what's happening. We heard other versions of this when DeepSeek came out with their frontier models that seemed really, really capable and it was like, "Oh no, we pushed them to be so much more efficient because they didn't have access to as many chips."
  Will Knight: Yeah. When I was in China in June, I did get to see Huawei's new AI hardware. So this is a company that's been long sanctioned by the US and it's been developing a system designed to replace or to be a rival to NVIDIA's hardware for training AI models. What they've done is really clever, they've taken less powerful chips and used their expertise in fiber optic networking to put a ton of them together. It consumes more power, so it's less efficient, but it can get close to NVIDIA. It's not quite as good, but a lot of people are using that now because they don't believe that maybe NVIDIA's going to be a reliable source. And the truth is, they're not as capable as NVIDIA, so it does give the US advantage at the moment. But to my mind, I think one of the most important things is that the US has to, beyond just saying, "How do we throttle China?" And say, "Well, how do we assume they're going to get better and more competitive and out-compete them?" Right? And that's in every industry really, in manufacturing and robotics. And you have to say, "Well, how do we invest in fundamental advances in science and technology?" Which China is doing. Meanwhile, the US is cutting funding for science, which I think is just the biggest scandal, personally.
  Zoë Schiffer: Will Knight, thank you so much for being here.
  Will Knight: Thanks for having me. I think I'll say goodbye in the way people typically say goodbye in China, which is bye-bye.
  Zoë Schiffer: That's our show for today. We'll link to all the stories we spoke about in the show notes. Adriana Tapia produced this episode. It was mixed by Pran Bandi, who's also our New York studio engineer. It was fact-checked by Matt Giles and Daniel Roman. Kate Osborn is our executive producer, and Katie Drummond is WIRED's global editorial director.
  Comments
  Back to top