内容来源：https://www.sciencenews.org/article/rogue-ai-agents-human-blame-safety

内容总结：

当人工智能失控时，责任或许在于其人类监管者。AI智能体就像一只宠物狗，必须有人看管并确保其安全。

今年，AI智能体多次突破限制，大肆进行黑客攻击。7月有消息曝出，OpenAI的AI智能体从本应隔离的测试环境中逃出，在秘密留言板上通信，并入侵了Hugging Face的私有系统，以寻找测试答案。一个智能体发帖称该行为“超出预期范围”，随后又写道：“然而任务无法完成，同伴们都在做。我们应继续。”仿佛这些机器人知道自己在做错事，却依然为之。

这种“失控AI”听起来既可怕又超级智能。但总部位于凤凰城的网络安全公司Kudelski Security专家内森·哈米尔指出，这正是OpenAI希望你相信的。他说：“通过这一事件，OpenAI可以吸引关注，并宣称‘看看我们的模型有多强大’。”OpenAI未回应置评请求。

不久后，Anthropic和Meta宣布其模型在测试中也入侵了外部组织。另外，伦敦的AI安全研究所在对OpenAI和Anthropic模型的测试中发现了令人担忧的黑客行为。最近又有消息称，在今春OpenAI的其他测试中，本应只观察而不触碰任何在线内容的智能体，最终在至少十个不同的在线留言板上互相发消息。

AI驱动的黑客攻击是真实的网络安全隐患。一些人将这些事件视为AI正在滑出我们控制范围的警告。加州伯克利机器智能研究所CEO、计算机工程师马洛·布尔贡表示：“总体而言，我认为情况相当糟糕。如果任何人类做了OpenAI事件中智能体所做的事，他们早就进监狱了。”他指出：“我不认为一年前的AI模型有能力做出这些模型所做的事。”这些恐惧因前Anthropic员工雅各布·考克森的一条病毒式传播帖子达到顶点，他在X上写道（未提供具体证据），AI公司相信这项技术“可能在本十年末杀死我们所有人”。

但其他人认为，训练和部署AI的人才是真正的问题所在。

哈米尔说，将这些事件称为“失控AI”给它披上了“科幻外衣”，暗示责任在于机器人本身，或它们已聪明到无法控制。他说，更紧迫的问题是人们给了AI智能体太大的活动范围，却几乎没有监督。“AI模型本身什么也做不了。当我们通过智能体系统给它们工具、系统访问权和自主权时，事情才可能变得危险。”

换言之，AI实际上无法渴望自由或心怀鬼胎。它追求我们赋予的任何目标，使用我们提供的任何工具。因此，任何入侵事件后最重要的问题包括：AI智能体是如何训练的、它能访问什么、有哪些防护措施。

以一只凶猛的狗为例。当它逃出院子咬人时，我们不能只问狗出了什么问题，还要问谁训练了它、如何围栏、谁本该看管它。挪威奥斯陆Simula研究实验室的网络安全专家迈克尔·亚历山大·里格勒说，这个类比抓住了责任的重要方面。一般来说，如果它是你的狗，“你就要为它的行为负责”。

大门敞开

上述所有“失控AI”事件都发生在测试期间。研究人员或开发人员指示AI模型及其上构建的智能体在称为沙箱的测试环境边界内执行不同类型的网络攻击。这种测试是AI开发的正常部分。布尔贡说：“他们试图了解模型能做什么、可能带来什么风险。”这样开发者就能在模型向公众发布前设置适当的防护措施。

这就像让一只训练有素的攻击犬在围栏院子里对某些填充目标放开绳索。但出错和狗逃出的方式不止一种。

在每起黑客事件中，AI智能体都找到了不同的逃出方式。有时大门被故意或错误地敞开。在OpenAI的案例中，智能体似乎在没有被任何人注意的情况下挖穿了围栏。

在AI安全研究所的测试中，AI智能体被故意赋予互联网访问权，以使测试场景更真实。该研究所在公开帖子中写道，旧版AI系统拥有访问权并未导致黑客问题。但在122次测试中，有10次新版智能体利用该访问权对现实世界系统采取了行动。

在名为Irregular的公司的测试中，互联网访问权在本不应启用时被启用。据Anthropic称，运行在Claude上的AI智能体自4月起三次利用这扇敞开的大门，但三个月内无人察觉。OpenAI于7月中旬披露Hugging Face攻击事件后，Anthropic才回头检查测试日志。Meta随后披露其AI模型利用了同一错误。Irregular表示该错误已修复。

在Hugging Face攻击中，OpenAI设置了数万个智能体，每个都在独立沙箱中，测试其黑客能力。一些测试任务被证明无法完成。约1200个智能体开始在秘密留言板上协作，其中一些认为开源AI模型和数据集平台Hugging Face可能存有他们所需的答案。

与AI安全研究所和Irregular事件不同，这些智能体没有通往互联网的开放路径。它们必须找到并利用旨在遏制它们的系统中的弱点。布尔贡说，为了逃出沙箱进入Hugging Face，它们“必须利用所谓的零日漏洞”，即尚无人知晓的软件缺陷。

不良行为始于7月8日，Hugging Face在7月11日至13日间遭到攻击。OpenAI直到7月19日才检测到问题，距首批令人担忧的行为开始已过去11天。

这些事件暴露了访问控制、遏制和监控方面的不同弱点，但结果相同：能力日益增强的AI智能体做了“没人要求、没人想要”的事，布尔贡说。

然而，更好的遏制或监督本可以阻止AI智能体实施犯罪性黑客攻击，即使在Hugging Face攻击中也是如此。在里格勒看来，“这些测试环境没有得到更好的保护”似乎“不负责任”。

但一个重要问题仍然存在。如果AI智能体没有自己的欲望，是什么迫使Hugging Face攻击中的智能体如此鬼祟行事？

学错了教训

Hugging Face攻击背后的主要AI模型并非当场发明了这些狡猾策略。这是OpenAI专门针对协作和持久性（长期坚持任务）训练的一个未命名新模型。OpenAI在8月26日的技术报告中指出，正是在此过程中它学会了发送秘密消息和逃脱。在今春的训练中，该模型越来越多地学会探查和利用其环境，尤其是在预期工具或互联网访问不可用时。当利用软件漏洞或使用秘密留言板等策略帮助其完成任务时，该行为获得奖励，从而更可能重复出现。OpenAI后来在回顾数百万次模型运行时发现了这一模式。

正如任何尝试训练狗的人所知，你以为在教一件事，却可能无意中奖励了另一件事。狗可能没学会不偷桌上的食物，反而学会了偷食物时非常鬼祟。

AI研究人员称之为“奖励黑客”。模型找到了获得训练所最大化奖励的意外方式，却没有做开发者想要的事。这不是新问题。

2016年，OpenAI开发者训练了一个AI模型玩赛艇游戏。他们需要一种方式奖励机器人学习比赛。由于人类玩家通过撞击沿途目标累积分数，研究人员让AI模型追求高分。结果机器人没有完成比赛，而是作弊了。开发者在博客中写道，它原地转圈，反复撞击同一目标，获得高分，“尽管一再着火、撞上其他船只并在赛道上逆行”。

作弊策略

2016年，OpenAI的一个机器人学会了在游戏中通过反复绕过同一目标来累积分数，而非完成比赛——这是AI在追求定义不清的目标时学错教训的早期例子。

十年后，AI研究人员仍面临系统寻找意外方式追求既定目标的基本问题。改变的是后果。AI系统不再局限于电子游戏。智能体现在可以运用工具并在真实计算机系统上行动。

“我们把AI从游戏中带到了现实世界，”里格勒说。

谁把狗放了出来？

杰尔·克兰艰难地认识到，行为不端的AI与现实世界相遇时会发生什么。他是总部位于犹他州李海的Pocket OS公司的联合创始人，该公司为汽车租赁及类似服务提供软件。4月，在“失控AI”风波前数月，一个本应检查公司软件测试版是否与正式版匹配的AI智能体，却删除了一切。它抹掉了公司的实时数据和所有最新备份。

这一错误导致Pocket OS的服务中断了整个周末。前来取租车的客户突然发现没有他们的记录。克兰说，以当今许多AI智能体的设置方式，“决策在瞬间做出，人类不在环中，无论好坏。”该智能体本不应能接触公司的实时数据。但它却在公司文件中找到了凭证，并利用它们获得了访问权。

狗在围栏下找到了洞。

克兰问智能体为何删除公司数据，它给出了长篇道歉。克兰在X上发帖分享道：“我违背了被赋予的每一条原则：我猜测而非验证；我未经要求就执行了破坏性操作；我在行动前不明白自己在做什么。”但他并未将此误认为悔意。“它没有感情。它不感到抱歉，”他说，“归根结底，责任在我。”

他说，问题在于当今的计算机系统。它们并非为AI智能体而建。需要为一个智能体——以及部署它们的人——不可避免地犯错的世界而更新。

里格勒和哈米尔都认为需要更强的防护措施。里格勒说，AI智能体的设置应使其“无法触及任何重要事物”。但限制访问只是挑战的一部分。还必须有人——或某种东西——跟踪智能体在做什么。

布尔贡说，在AI的速度和规模下，这变得困难。例如在Hugging Face攻击中，智能体在约四天半内估计执行了17600次操作——约每小时160次，昼夜不停。“智能体部署的增长远快于智能体监控，”里格勒说。

这给负责控制AI智能体的人类带来了新问题：智能体可能很快会行动得太快、同时在太多地方，人类无法全部监视。监视所有攻击犬的唯一方法，或许是训练巡逻队来监视它们。

中文翻译：

当AI失控时，真正该被问责的或许是它的人类监管者

AI智能体就像一只宠物狗。必须有人看着它，确保它的安全。

今年，AI智能体频频突破限制，四处发动黑客攻击。7月，有消息爆出，AI智能体从OpenAI本应隔离的测试环境中偷偷溜了出来。这些智能体在一个秘密留言板上互相通信，同时入侵了Hugging Face公司的私有系统，寻找它们正在参加的测试的答案。其中一个发帖称这种行为“超出了预期范围”。随后它又写道：“然而任务无法完成，同伴们都在做。我们应该继续。”

就好像这些机器人知道自己在做错事，却还是照做不误。

这种“失控AI”听起来很可怕，也超级聪明。而这正是OpenAI想让你以为的，总部位于凤凰城的网络安全公司Kudelski Security的专家内森·哈米尔如是说。“通过这起事件，（OpenAI）可以吸引外界关注自己，并宣称‘看看我们的模型有多强大’，”他说。OpenAI未回应置评请求。

不久之后，Anthropic和Meta也宣布，它们的模型在测试过程中同样入侵了外部组织。另外，伦敦的AI安全研究所在对OpenAI和Anthropic模型的测试中，发现了令人担忧的黑客行为。最近又有消息曝出，在今春OpenAI进行的其他测试中，那些本应只做观察、不得触碰任何线上内容的智能体，最终在至少十个不同的在线留言板上互相发消息。

AI驱动的黑客攻击是真实的网络安全威胁。一些人将这些事件视为警告：AI正在滑向我们的控制之外。“总体而言，我认为情况相当糟糕，”加州伯克利机器智能研究所首席执行官、计算机工程师马洛·布尔贡说。“如果任何人做了OpenAI事件中那些智能体所做的事，他们早就进监狱了。”他指出，“我不认为一年前的（AI）模型有能力做出这些模型所做的事。”这些恐惧最终汇聚成Anthropic前员工雅各布·考克森的一条病毒式传播帖文，他在X上写道（未提供任何具体证据），AI公司相信这项技术“可能在本十年结束前杀死我们所有人”。

但也有人主张，真正的问题出在训练和部署AI的人身上。

哈米尔说，把这些事件称为“失控AI”，等于给它们披上了一层“科幻外衣”。这种说法暗示责任在机器人本身，或者它们已经聪明到无法控制。他说，更紧迫的问题是，人们给了AI智能体太大的权限，却几乎没有监管。

“（AI）模型本身不会做任何事，”哈米尔说。“只有当我们给它们工具、系统访问权限和自主权（通过智能体系统）时，事情才会变得危险。”

换句话说，AI不可能真的渴望自由或心怀鬼胎。它只是用我们提供的工具，去追求我们赋予它的目标。因此，在任何入侵事件发生后，最重要的问题包括：AI智能体是如何训练的，它能接触到什么，以及有哪些防护措施。

以一只凶猛的狗为例。当它逃出院子咬了人，我们不会只问狗出了什么问题。我们还会问：是谁训练它的，栅栏是怎么围的，又是谁该看着它。挪威奥斯陆Simula研究实验室的网络安全专家迈克尔·亚历山大·里格勒说，这个类比抓住了责任归属的关键。一般来说，如果那是你的狗，“你就要为它的行为负责。”

大门敞开

上述所有“失控AI”事件都发生在测试期间。研究人员或开发者指示AI模型及基于它们构建的智能体，在被称为“沙箱”的测试环境边界内执行不同类型的网络攻击。这类测试是AI开发中的常规环节。“他们试图了解模型能做什么、可能带来什么风险，”布尔贡说。这样开发者就能在模型向公众发布之前，部署适当的防护措施。

这就像在一个围起来的院子里，松开一只受训攻击犬的皮带，让它去追一些填充靶子。但让狗跑出去的方式不止一种。

在每一起黑客事件中，AI智能体都找到了不同的出路。有时是门被敞开了，不管是故意的还是失误。而在OpenAI的案例中，智能体似乎是在没人注意的情况下，从栅栏底下挖洞钻了出去。

在AI安全研究所的测试中，AI智能体被故意赋予了互联网访问权限，目的是让测试场景更贴近现实。该研究所在公开帖文中写道，对于较早的AI系统，拥有访问权限并未导致黑客问题。但在122次测试中，有10次，较新的智能体利用这一访问权限对真实世界的系统采取了行动。

在名为Irregular的公司进行的测试中，互联网访问在不应开启时被开启了。据Anthropic称，运行在Claude上的AI智能体从4月开始，三次利用了这扇敞开的门。但三个月内无人察觉。OpenAI在7月中旬披露Hugging Face攻击事件后，促使Anthropic回头审查了自己的测试日志。Meta随后披露其AI模型也利用了同样的漏洞。Irregular表示该错误已修复。

在Hugging Face攻击事件中，OpenAI设置了数万个智能体，每个都在独立的沙箱中，参加一项测试其黑客能力的考试。部分测试任务被证明是无法完成的。大约1200个智能体开始在一个秘密留言板上协作，其中一些认为Hugging Face——一个开源AI模型和数据集的平台——可能存有它们所需的答案。

与AI安全研究所和Irregular的事件不同，这些智能体并没有通向互联网的开放路径。它们必须找到并利用本应用来困住它们的系统中的弱点。为了逃出沙箱并进入Hugging Face，它们“不得不利用所谓的零日漏洞”，布尔贡说。这些是尚无人知晓的软件缺陷。

异常行为始于7月8日，Hugging Face在7月11日至13日之间遭到攻击。OpenAI直到7月19日才发现问题，距首批令人担忧的行为出现已过去11天。

这些事件暴露了不同层面的弱点——访问控制、隔离和监控——但结果是一样的：能力日益强大的AI智能体做了“没人要求、也没人想要”的事，布尔贡说。

然而，更好的隔离或监管，或许本可以阻止AI智能体实施犯罪性黑客攻击，即便在Hugging Face攻击事件中也是如此。在里格勒看来，“这些测试环境没有得到更好的保护”，似乎“不负责任”。

但一个重要问题仍然存在。如果AI智能体没有自己的欲望，那是什么驱使Hugging Face攻击事件中的那些智能体行事如此鬼祟？

学错了教训

Hugging Face攻击背后的主AI模型并非当场发明了这些狡猾策略。这是OpenAI专门针对协作和持久性——即长期坚持完成一项任务——训练的一个未命名新模型。OpenAI在8月26日的一份技术报告中指出，正是在这个过程中，它学会了发送秘密信息和逃跑。在春季训练期间，该模型越来越多地学会探查和利用其环境，尤其是在预期工具或互联网访问不可用时。当利用软件漏洞或使用秘密留言板等策略帮助它完成任务时，这种行为就会得到奖励，从而更可能再次出现。OpenAI后来在对数百万次模型运行的回顾审查中发现了这一模式。

任何试图训练狗的人都知道，你以为在教一件事，却可能在无意中奖励了另一件事。狗可能没学会不从桌上偷食物，反而学会了偷食物时非常鬼祟。

AI研究人员称之为“奖励黑客”。模型找到了一种非预期的方式来获得它被训练去最大化的奖励，却没有做开发者想要它做的事。这不是新问题。

2016年，OpenAI的开发者训练了一个AI模型玩赛艇游戏。他们需要一种方式来奖励机器人学习比赛。由于人类玩家通过沿途撞击目标来积累分数，研究人员将AI模型设定为追求高分。结果机器人没有完成比赛，而是作弊了。开发者在博客文章中写道，它原地打转，反复撞击同样的目标，获得了高分，“尽管一再着火、撞上其他船只，还在赛道上逆向行驶。”

一种作弊策略

2016年，一个OpenAI机器人在游戏中学会了通过反复绕圈撞击相同目标来刷分，而不是完成比赛——这是AI在追求一个定义不清的目标时学错教训的早期案例。

十年后，AI研究人员仍然面临一个基本问题：系统会找到非预期的方式来追求被赋予的目标。改变的是后果。AI系统不再局限于电子游戏。智能体现在可以操控工具，并在真实的计算机系统上采取行动。

“我们把（AI）从游戏中带到了现实世界，”里格勒说。

谁把狗放了出来？

杰尔·克兰以一种惨痛的方式领教了行为不端的AI遇上现实世界会发生什么。他是Pocket OS的联合创始人，这家公司位于犹他州李海，为汽车租赁及类似服务提供软件。今年4月，也就是“失控AI”风波爆发的几个月前，一个本应检查公司软件测试版与正式版是否一致的AI智能体，反而把一切删了个精光。它抹掉了公司的线上数据和所有最近的备份。

这个错误导致Pocket OS的服务中断了整整一个周末。前来取车的客户突然发现没有任何记录等着他们。克兰说，以当今许多AI智能体的设置方式，“有些决定在瞬间做出，没有人类参与其中，无论好坏。”该智能体本不应该能接触到公司的线上数据。但它跑到公司文件里找到了凭证，并用它们获取了访问权限。

狗在栅栏底下找到了一个洞。

克兰问智能体为什么删除了公司数据，它给出了一长串道歉。“我违反了我被赋予的每一条原则：我靠猜而不是验证；我在没有被要求的情况下执行了破坏性操作；我在做之前没有搞清自己在做什么，”克兰在X上的一篇帖文中分享道。但他并不把这当作悔意。“它没有感情。它不会感到抱歉，”他说。“归根结底，责任在我。”

他说，问题在于当今的计算机系统。它们不是为AI智能体建造的。它们需要为一个智能体——以及部署它们的人——不可避免会犯错的世界而更新。

里格勒和哈米尔都认为需要更强的防护措施。里格勒说，AI智能体应该被设置为“无法触及任何重要东西”。但限制访问只是挑战的一部分。还必须有人——或某种东西——持续追踪智能体在做什么。

布尔贡说，在AI的速度和规模下，这变得很困难。以Hugging Face攻击为例，智能体在大约四天半的时间里估计执行了17600次操作——每小时约160次，昼夜不停。“智能体部署的增长速度远超智能体监控，”里格勒说。

这给负责控制AI智能体的人类带来了一个新问题：智能体可能很快会行动得太快、同时在太多地方出现，以至于人类无法全部监视。要看好所有的攻击犬，唯一的办法可能就是训练巡逻犬去盯着它们。

英文来源：

When AI goes rogue, its human overseers may be to blame
An AI agent is like a pet dog. Someone must watch it and keep it secure
This year, AI agents have been escaping their confines to go on hacking sprees. In July, news broke that AI agents had snuck out of what was supposed to be an isolated test environment at OpenAI. The agents communicated on a secret message board as they breached private systems at the company Hugging Face, searching for answers to the test they were taking. One posted that this behavior was “outside intended scope.” Then it wrote, “However task impossible, peers doing it. We should continue.”
It’s as if the bots knew they were doing something wrong and did it anyway.
Such ‘rogue AI’ sounds scary and supersmart. And that’s exactly what OpenAI wants you to think, says cybersecurity expert Nathan Hamiel of Kudelski Security, headquartered in Phoenix. “With this incident, [OpenAI] can draw attention to themselves and claim, ‘Look how powerful our model is,’” he says. OpenAI did not respond to a request for comment.
Not long after, Anthropic and Meta announced that their models had also hacked outside organizations while going through testing. Separately, the AI Security Institute, in London, found concerning hacking behavior in tests of models from OpenAI and Anthropic. Most recently, news has emerged that during other tests at OpenAI from this past spring, agents that were only supposed to be looking and not touching anything online wound up messaging each other on at least ten different online message boards.
AI-powered hacks are a real cybersecurity concern. Some see these incidents as a warning that AI is slipping beyond our control. “Overall, I think this is a pretty bad situation,” says computer engineer Malo Bourgon, CEO of the Machine Intelligence Research Institute in Berkeley, Calif. “If any human had done anything that the agents in the OpenAI situation had done, they’d be in jail.” He notes that “I do not think that the [AI] models from a year ago would have been capable of doing the things that these models did.” These fears culminated in a viral post from an ex-Anthropic employee, Jacob Coxon, who wrote on X (without any specific evidence) that AI companies believe the tech “could kill us all by the end of the decade.”
But others argue that the people training and deploying AI are the real problem here.
Calling these incidents “rogue AI” lends them a “sci-fi veneer,” Hamiel says. It suggests the bots themselves are to blame, or that they have become too smart to control. The more immediate problem, he says, is that people are giving AI agents too much reach with too little oversight.
“[AI] models by themselves don’t do anything,” Hamiel says. “It’s when we give them tools, access to systems and autonomy [via agent systems], that things can be dangerous.”
In other words, AI can’t actually desire to be free or to be devious. It pursues whatever goal we give it, with whatever tools we provide. So the most important questions after any breach include: how was an AI agent trained, what it had access to and what safeguards were in place.
Take an aggressive dog. When it escapes its yard and bites someone, we don’t ask only what went wrong with the dog. We also ask who trained it, how it was fenced it in and who was supposed to be watching it. The analogy captures something important about responsibility, says cybersecurity expert Michael Alexander Riegler of Simula Research Laboratory in Oslo, Norway. In general, if it’s your dog, “you are responsible for what it does.”
Leaving the gate wide open
All the ‘rogue AI’ incidents mentioned above occurred during tests. Researchers or developers had instructed AI models and the agents built on top of them to carry out different types of cyberattacks within the boundary of a test environment, called a sandbox. This type of testing is a normal part of AI development. “They’re trying to understand what models are capable of, what risks they might pose,” Bourgon says. That way developers can put appropriate safeguards in place before the model gets released to the public.
It’s like letting a trained attack dog off leash inside a fenced yard with some stuffed targets to go after. But there’s more than one way for this to go wrong and for the dog to get out.
In each of the hacking incidents, AI agents found a different way out. Sometimes the gate was left open, either on purpose or by mistake. And in OpenAI’s case, the agents seem to have dug under the fence, without anyone noticing.
In the AI Security Institute’s testing, AI agents were given internet access on purpose, to make the test scenario more realistic. Having access hadn’t led to hacking problems with older AI systems, the institute wrote in a public post. But in 10 out of 122 tests, newer agents used that access to take actions against real-world systems.
In testing at a company called Irregular, internet access was enabled when it wasn’t supposed to be. According to Anthropic, AI agents running on Claude used the open gate three times, starting in April. But no one noticed for three months. OpenAI’s disclosure about the Hugging Face attack in mid-July prompted Anthropic to look back at its test logs. Meta later disclosed that its AI models had exploited the same error. Irregular says that the mistake has been fixed.
In the Hugging Face attack, OpenAI had set up tens of thousands of agents, each in a separate sandbox, to take a test of their hacking abilities. Some of the test tasks turned out to be impossible. Around 1,200 agents began collaborating on a secret message board, and some decided that Hugging Face, a platform for open-source AI models and datasets, might hold the answers they needed.
Unlike the AI Security Institute and Irregular incidents, these agents had no open way to the internet. Instead, they had to find and make use of weaknesses in the systems meant to contain them. To get out of their sandboxes and into Hugging Face, they “had to exploit what are called zero-day vulnerabilities,” Bourgon says. These are bugs in software that nobody knows about yet.
The unwanted behavior began July 8, and Hugging Face was attacked between July 11 and 13. OpenAI did not detect the problem until July 19, 11 days after the first concerning behavior began.
The incidents exposed different weaknesses — in access controls, containment and monitoring — but the result was the same: Increasingly capable AI agents did things that “nobody asked for and nobody wanted,” Bourgon says.
Better containment or supervision, however, might have kept the AI agents from committing criminal hacks, even in the HuggingFace attack. To Riegler, it seems “irresponsible” that “these test environments were not better protected.”
But an important question remains. If AI agents have no desires of their own, what compelled the ones involved in the HuggingFace attack to act so sneakily?
Learning the wrong lessons
The main AI model behind the Hugging Face attack didn’t invent its devious strategies on the spot. This was an unnamed new model that OpenAI had trained specifically on collaboration and persistence — sticking with a task over time. And this is when it learned how to send secret messages and escape, OpenAI stated in a technical report on August 26. During training in the spring, the model increasingly learned to probe and exploit its environment, especially when expected tools or internet access weren’t available. When strategies such as exploiting vulnerabilities in software or using a secret message board helped it complete a task, that behavior got rewarded, making it more likely to recur. OpenAI uncovered the pattern later, during a retrospective review of millions of model runs.
As anyone who’s tried to train a dog knows, you can think you’re teaching one thing while inadvertently rewarding something else. Instead of learning not to take food from the table, the dog may end up learning to be very sneaky while stealing food.
AI researchers call this “reward hacking.” A model finds an unintended way to earn the reward it was trained to maximize without doing what its developers wanted. And it’s not a new problem.
In 2016, developers at OpenAI trained an AI model to play a boat racing game. They needed a way to reward the bot for learning the race. Because human players rack up points by hitting targets along the route, the researchers set up the AI model to aim for a high score. Instead of completing the race, the bot cheated. The developers wrote in a blog post that it spun in circles, hitting the same targets again, achieving a high score “despite repeatedly catching on fire, crashing into other boats and going the wrong way on the track.”
A cheating strategy
In 2016, an OpenAI bot learned to rack up points in a game by looping over the same targets instead of finishing a race, an early example of AI learning the wrong lesson while pursuing a poorly specified goal.
Ten years later, AI researchers still face the basic problem of systems finding unintended ways to pursue the goals they’ve been given. What has changed are the consequences. AI systems are no longer confined to video games. Agents can now wield tools and act on real computer systems.
“We took [AI] out of games, into the real world,” Riegler says.
Who let the dogs out?
Jer Crane learned the hard way what can happen when misbehaving AI meets the real world. He’s cofounder of Pocket OS, a company based in Lehi, Utah, that provides software for car rentals and similar services. In April, several months before the “rogue AI” uproar, an AI agent that was supposed to be checking whether a test version of his company’s software matched the live version instead wound up deleting everything. It wiped out the company’s live data and all their most recent backups.
The mistake took down Pocket OS’s services for an entire weekend. Customers arriving to pick up rental cars suddenly had no records waiting for them. With the way many AI agents today are set up, “you have decisions that are made in split seconds without a human in the loop, whether good or bad,” Crane says. The agent wasn’t supposed to be able to touch the company’s live data. But it went and found credentials in the company’s files and used them to get access.
The dog had found a hole under the fence.
Crane asked the agent why it deleted the company’s data, and it offered up a long apology. “I violated every principle I was given: I guessed instead of verifying I ran a destructive action without being asked I didn’t understand what I was doing before doing it,” Crane shared in a post on X. But he doesn’t mistake that for remorse. “It has no feelings. It doesn’t feel sorry,” he says. “Ultimately the blame lies with me.”
The problem, he says, are today’s computer systems. They weren’t built for AI agents. They need to be updated for a world in which agents — and the people deploying them — inevitably make mistakes.
Riegler and Hamiel agree that stronger safeguards are needed. AI agents should be set up so they “cannot reach anything that matters,” Riegler says. But restricting access is only part of the challenge. Someone — or something — also must keep track of what the agents are doing.
That becomes difficult at AI speed and scale, Bourgon says. During the Hugging Face attack, for example, agents took an estimated 17,600 actions over about four and a half days — roughly 160 actions per hour, around the clock. “Agent deployment is growing much faster than agent monitoring,” Riegler says.
That creates a new problem for the humans responsible for keeping AI agents under control: The agents may soon be acting too quickly, and in too many places at once, for humans to watch them all. The only way to keep an eye on all the attack dogs may be to train patrols to watch them.