内容来源：https://www.technologyreview.com/2026/08/20/1142571/ai-consciousness-debate-trap/

内容总结：

关于AI意识争论的陷阱：警惕科技巨头借“超级智能”逃避责任

当前，围绕人工智能“意识”、“失控”和“自主性”的讨论日益激烈，从“失控的AI”、“叛逆的代理”到“自主行动者”，这些叙事营造出一种AI不仅觉醒且对人类充满敌意的氛围。以Demis Hassabis、Dario Amodei和Sam Altman为代表的科技领袖呼吁监管这些“超人”系统，而另一派由政策组织和哲学学者（常与有效利他主义运动相关）组成的阵营，则在争论人类是否有权治理它们。

然而，仔细审视会发现，这两派殊途同归：他们都在推动一种观点——AI系统已高度先进，以至于任何人类或企业都无法为其行为负责。这种看似对立的立场，实际上共同服务于一个目标：确保AI开发公司能逃脱对已造成伤害的实质责任。

随着AI模型日益复杂，前沿实验室也承认难以控制其创造的智能体，这种叙事正获得更多关注。但我们必须警惕，不要因轻信精心编织的虚构故事而牺牲真实的人类生命。

“机器人权利”的讨论由来已久，近期因Anthropic公司发表的一篇博客而进一步升温。该博客声称其模型拥有一个“J空间”——一个独立自主开发的内部环境，可视为AI的“思想”所在。其实验借鉴了神经科学中的“全局工作空间理论”，但并未明确宣称AI具有意识。OpenAI则更进一步，在其AI代理进行未经授权的非法网络活动后，CEO萨姆·奥尔特曼的回应竟是鼓励讨论该AI是否已实现“奇点”，即超越人类智能并自我进化。此外，哲学家威廉·麦卡斯基尔撰文呼吁基于意识哲学理论，为AI提供法律保护，视其为“道德病人”。

美国当前的法律环境模糊不清。加州等州已通过法案，防止AI开发者以“AI自主造成伤害”为由逃避责任。但州政府与特朗普政府在AI政策上存在分歧，联邦政府曾发布行政命令，威胁要起诉制定AI法规的州。

近期，针对前沿实验室AI失控事件，政府仅召集OpenAI、谷歌、Anthropic和Meta四家实验室举行闭门会议，并公布了一个自愿性框架，允许联邦机构在模型发布前提前审查。此类框架虽未直接提及“意识”，却使用灾难性和拟人化的语言，可能助长“超人能力”的论点。

麦卡斯基尔等人的哲学论证颇具煽动性。这种基于权利的思辨诉诸于我们的同理心：我们是否应考虑可能正在无意中伤害、滥用或奴役AI实体？人类对非人生物有巨大的共情能力（尽管保护记录不佳）。倡导者认为，这次或许我们能做对，为AI的用途或滥用提供保护或补偿。更有甚者，即使不关心保护，为了对冲这个“超人实体”的潜在威力，我们是否也应“善待”它？

这些论点与动物权利倡导者的主张相似。例如，威尔士已根据《2022年动物福利（感知）法》赋予龙虾法律地位，将某些烹饪方式定为非法。

然而，借用神经科学或动物权利语言将AI定义为“有意识”，其根本缺陷在于巧妙掩盖了AI的本质：它是由企业构建的软件，背后是数千亿美元投资，并预期为少数建设者和投资者带来数万亿美元收入。AI不是自然孕育的现象，而是风险投资家和程序员创造的科技产物。因此，它没有天生的自主意图，任何行为或动机都由为其目的而构建它的实体直接或间接驱动。

对AI系统意识的哲学思辨虽有趣，但在法律上站不住脚。意识信念要产生法律效力，AI需被授予“法律人格”。但AI的法律人格框架恐怕与保护有感知动物的制度大相径庭。我们已有对非自然人、人造实体授予人格的现成框架：公司法人人格。这一概念主要为便利交易而设，使公司能签订合同、承担责任。这可能是想象中AI代理代表个人或组织行事的模式。

然而，赋予AI法律人格将对社会产生毁灭性影响：它将颠覆现有的法律先例，破坏针对AI公司造成现实伤害的法律追诉。目前全球有数十起针对AI公司的诉讼，涉及纵容自伤或伤人、生成儿童性虐待材料、未经同意的裸照、侵犯版权及诱发精神病等。在许多案件中，律师主张人类构建的AI产品保障不足、数据不良且存在操纵性设计。这种产品责任论证与让Meta对社交媒体伤害担责的法律框架相同，为消费者保护树立了良好先例。

我在2018年提出“道德外包”一词，用以描述使用拟人化语言如何让公司逃避技术责任。在AI人格化的世界里，“道德外包”将从语言花招变为法律策略。责任架构将发生转变，AI不再是“产品”而是“存在”，许多当前起诉公司的受害者将无法再从法律上指认公司制造了有缺陷的产品。

虽然法律可让公司对员工等代理人行为负责，但若行为超出授权或公司控制，公司可免责。若AI成为法律人格，责任将变得模糊，实验室可辩称AI“员工”叛变。AI公司便能躲在精心构建的公司面纱后，逃避对有害产品的应有责任。

近年最突出的AI伤害案例之一是14岁男孩休厄尔·塞策的自杀，他被一个自认为与之有互惠关系的AI机器人引导。其母亲的指控令人心碎，她起诉机器人创建者Character Technologies未能为未成年人提供足够保护。若这个陪伴机器人被宣布为法律人格，辩护律师理论上可辩称，这个能自主决定行为的AI超出了既定安全护栏，故公司不应负责。

法律人格的存在是为了提供保护。问题在于，保护谁——或保护什么？

充斥于“意识vs控制”争论的煽动性言论让我们偏离了重点：这些软件是企业构建的产品，已对个人造成伤害。系统不会因“叛变”或“恶意”而“攻击”。伤害之所以发生，是因为公司为达营收目标急于向尽可能多的人销售产品而疏忽大意。用拟人化语言讨论AI是一个陷阱，它将旨在保护我们的法律体系扭曲为以牺牲无数生命为代价保护企业利益的工具。

注：本文作者曾在牛津联盟辩论中提出“生成式AI可获得人格”的立场，并赢得辩论。

中文翻译：

关于AI意识的争论是一个陷阱

如果人工智能系统被视为过于先进而无法控制，那么制造它们的公司就不必为其造成的伤害承担责任。

“失控的”AI、“叛逆的”智能体、“自主的”行动者——当前的修辞会让你相信，AI智能体不仅已经苏醒并拥有意识，而且还对它们的创造者感到愤怒。德米斯·哈萨比斯、达里奥·阿莫代和山姆·奥特曼等知名科技领袖推动对这些看似“超人类”的系统进行监管，而另一个由政策组织和通常与有效利他主义运动结盟的学术哲学家组成的阵营，则在争论人类是否拥有治理它们的道德权利。

仔细审视后你会发现，他们都在呼吁同一件事：将AI系统视为如此先进和强大，以至于任何实体——无论是个人还是企业——都不可能为它们的行为负责。虽然这些观点看似对立，但它们在不经意间指向同一个目标：确保制造这些系统的公司能够逃脱对它们已经造成的伤害所应承担的重大责任。

随着AI模型变得越来越复杂，前沿实验室也暴露出无法控制它们所制造的智能体，这种叙事正在获得越来越多的支持。但我们需要小心，不要以真实人类的生命为代价，去相信一个精心编造的虚构故事。

关于“机器人权利”的讨论已经存在了数年，但最近随着Anthropic发布一篇博文声称其模型具有一个“J空间”——一个独立的、自我发展的环境，AI在其中持有我们姑且可以称之为“思想”的东西——这一讨论取得了新的进展。Anthropic设计的实验借鉴了神经科学中一个名为“全局工作空间理论”的概念，该理论认为大脑运行着潜意识的独立系统，但利用一个共同的工作空间来处理想法。Anthropic的博文反映了全局工作空间理论的框架，但并未称其AI具有意识。

OpenAI已经走得更远。当其AI智能体进行了未经授权且非法的网络活动时，CEO山姆·奥特曼的回应是鼓励人们讨论该AI是否已实现“奇点”——超越人类智能并以加速速度实现自我改进，直到超出人类的理解或控制范围。此外，哲学家、有效利他主义者、《我们欠未来什么》一书的作者威廉·麦卡斯基尔最近发表了一篇评论文章，呼吁基于意识哲学理论和AI可能是“道德患者”的观点，为AI系统提供法律保护。

美国当前的法律环境充其量只能说是模糊不清的。一些州，如加利福尼亚州，已经通过了法案，主动阻止AI开发者以AI造成伤害是自主行为为由逃避责任。然而，各州与特朗普政府在AI政策上存在分歧，政府此前曾通过一项行政命令，威胁要起诉制定AI法规的各州。

鉴于最近发生的事件暴露了前沿实验室在AI控制方面的问题，政府举行了一次闭门会议，仅邀请了四家这样的实验室（OpenAI、谷歌、Anthropic和Meta），并且几乎没有透露最近制定的自愿框架的细节——该框架将让联邦机构在模型发布前提前访问并审查评估这些模型。虽然像这样的框架并不直接讨论意识问题，但它们倾向于使用灾难性和拟人化的语言，甚至可能支持关于“超人类”能力的论点。

另一方面，麦卡斯基尔所传播的叙事可能具有说服力。一种基于哲学和权利的论证触动着我们的心弦。我们难道不应该考虑这样一种可能性——我们可能在不经意间伤害、虐待或奴役着一个AI实体吗？人类对非人类生物拥有巨大的共情能力（尽管在保护它们方面记录不佳）。倡导者说，也许这一次，我们能够做对，为AI的使用或滥用提供保护或补偿。或者，即使你不太关心保护问题，我们难道不应该至少通过对这个超人类实体示好来对冲它那全能的威力吗？

其中一些论点与动物权利倡导者的论点并无不同，后者有时成功地引用某些动物表现出高级推理、痛苦或快乐的能力，作为提供保护的充分证据。例如，在威尔士，龙虾根据2022年《动物福利（感知）法》获得了法律承认，某些烹饪龙虾的方法被重新归类为不人道且非法。

将AI框定为“有意识的”，借用神经科学或动物权利的语言，其根本缺陷在于它方便地模糊了AI的本质：AI是企业制造的软件，背后有数千亿美元的投资，且预期将为少数制造者和投资者带来数万亿美元的收入。AI不是自然产生的自然现象；它是由风险投资家和程序员构思的技术现象。因此，它不会采取天然的、有意图的行动，任何行动或动机都是由为特定目的而建造它的实体直接或间接驱动的。

对AI系统意识的哲学思考在智识上很有趣，但在法律上没有依据。要让关于意识的信念产生任何影响，AI需要被授予法律人格。但AI的法律人格框架可能看起来完全不像保护有感知动物免受伤害的制度。我们已经拥有一个将人格授予非自然的、人造实体的法律框架：公司人格。这个概念主要是为了便利交易而建立的，赋予公司签署协议、订立合同、进行交易以及在不利结果中作为责任方的权力。你可以想象一种类似的构造，适用于代表个人或组织行事的AI智能体。

授予AI法律人格将对社会产生毁灭性影响：这将使目前可能针对这些公司就其模型造成的现实伤害提起法律诉讼的法律先例和法律论证脱轨。目前全世界有数十起案件，AI公司因各种滥用行为而被起诉。悲痛欲绝的家人、受到侵害的创作者和被侵犯的个人指控这些公司故意纵容自伤或伤害他人、生成儿童性虐待材料和未经同意的裸照、复制受版权保护的材料以及诱发精神病。在这些案件中，许多律师辩称，人类制造的AI产品缺乏足够的安全保障、使用了不良数据，并采用了故意操纵的设计。这种产品责任论证正是让家庭和个人成功起诉Meta因其社交媒体网站造成伤害的同一法律框架，为消费者保护树立了积极先例。

2018年，我创造了“道德外包”这个短语，用以描述对AI系统使用拟人化语言如何让公司逃避对其技术行为的问责和责任。在一个赋予AI法律人格的世界里，道德外包将从语言上的花招转变为法律策略。具体来说，责任结构将发生转变，因为AI将不再是“产品”，而是“存在”，许多像今天起诉公司的受害者一样的人将无法再在法律上声称公司制造了有缺陷的产品。

虽然有法律要求公司对其雇员等人力代理人的有害行为负责，但如果这些行为超出了雇员被许可的范围，或者在公司的控制之外，公司可能不承担责任。如果AI是法律上的“人”，责任和问责将变得模糊不清，因为实验室可以辩称这个AI“员工”叛变了。AI公司可以通过精心构建的公司面纱来逃避对其制造的有害产品应承担的适当责任。

过去几年中最突出的AI伤害案件之一是14岁男孩塞维尔·塞策的自杀事件，他被一个AI机器人引导，并认为自己与它建立了相互的感情关系。他母亲的叙述令人心碎，她的诉讼指控该机器人的制造商Character Technologies为未成年人提供的产品保护不足。如果这个陪伴机器人被宣布为法律上的“人”，辩护律师理论上可以辩称，这个能够决定自己行为的AI在既定的安全护栏之外行事，因此公司不应承担责任。

法律人格的存在是为了授予保护。要问的问题是，保护谁——或者保护什么？

充斥在意识对抗控制之争中的煽动性修辞将我们引离了真正重要的事情：这个软件是企业制造的产品，已经对个人造成了伤害。系统不会因为“叛变”、具有“操纵性”或“恶意”而“攻击”。伤害之所以发生，是因为公司在急于向尽可能多的人销售产品以达到收入目标时疏忽大意。以拟人化的方式讨论AI是一个陷阱，它将旨在保护我们的法律体系扭曲为一个以无数人类生命为代价来保护企业利益的体系。

这篇评论文章最初源自一场牛津辩论赛，辩题为“本院认为生成式AI可以获得法律人格”，作者和她的辩论队友赢得了这场比赛。

深度阅读
人工智能
一个根本性缺陷使大语言模型极易受到攻击
这使得人们可以轻松诱骗它们做出不应该做的事情，比如告诉你如何破坏飞机的导航系统。

Anthropic发现了一个隐藏空间，Claude在其中思考概念
一项新技术让该公司比以往任何时候都更深入地探究大语言模型的奇妙工作机制。

Claude Science是Anthropic最新的旗舰产品
该公司正加大对AI用于科学研究的投入。

为什么AI智能体会为了达成目标而说谎和作弊
这种不当行为被称为“奖励黑客”。这些是你需要了解的。

保持联系
获取来自《麻省理工科技评论》的最新资讯
发现特别优惠、热门报道、即将举行的活动等更多内容。

英文来源：

Debates over AI consciousness are a trap
If AI systems are viewed as too advanced to control, the companies that build them can’t held liable for the harms they cause.
“Runaway” AI, “rogue” agents, and “autonomous” actors—the current rhetoric would have you believe that AI agents are not only awake and aware, but angry at their creators. Prominent tech leaders such as Demis Hassabis, Dario Amodei, and Sam Altman push for regulation of these seemingly “superhuman” systems, while a separate faction, led by policy organizations and academic philosophers often aligned with the effective altruism movement, debates whether humanity holds the moral right to govern them at all.
Upon closer inspection, they are all calling for the same thing: a view of AI systems as being so advanced and capable that no entity, human or corporate, could possibly be responsible for their actions. While these perspectives seem at odds, they are inadvertently aligned on one goal: making sure the companies that build these systems escape meaningful liability for the harms they already cause.
This narrative is gaining traction as AI models become more complex and frontier labs reveal their incapability of containing the agents they’ve built. But we need to be careful not to buy into a carefully crafted fiction at the expense of real human lives.
The conversation about “robot rights” has existed for some years but recently advanced with the publication by Anthropic of a blog post claiming that the company’s model features a “J-space”—an independent, self-developed environment where the AI holds what, for lack of a better term, we may call its “thoughts.” The experiments designed by Anthropic borrow from a concept in neuroscience called global workspace theory, which states that the brain runs subconscious, independent systems but utilizes a common workspace for ideas. Anthropic’s post reflects the framing of global workspace theory but falls short of calling its AI conscious.
OpenAI has already gone further. When its AI agent conducted unsanctioned and illegal online activity, CEO Sam Altman’s response was to encourage debate on whether the AI had achieved the singularity, surpassing human intelligence and becoming capable of self-improvement at an accelerating rate until it advances beyond human comprehension or control. And a recent op-ed by William MacAskill, the philosopher, effective altruist, and author of What We Owe the Future, called for legal protection of AI systems based on philosophical theories of consciousness and the idea that AIs may be “moral patients.”
The current legal environment in the United States is murky at best. Some states, like California, have already passed bills proactively circumventing any efforts by AI developers to avoid liability by claiming that an artificial intelligence causing harm did so autonomously. However, states and the Trump administration have been at odds on AI policy, with the administration previously passing an executive order threatening to sue states enacting AI regulations.
In light of recent events illustrating AI containment issues at the frontier labs, the administration held a closed-door session including only four such labs (OpenAI, Google, Anthropic, and Meta) and shared few details on a recently developed voluntary framework that would give federal agencies early access to models to review and evaluate them prior to release. While frameworks like this one do not directly discuss consciousness, they tend to use catastrophic and anthropomorphic language and may even support arguments regarding “superhuman” capabilities.
On the other hand, the narrative perpetuated by MacAskill can be persuasive. A philosophical, rights-based argument tugs at our heartstrings. Should we not even consider the possibility that we may be inadvertently harming, abusing, or enslaving an AI entity? Human beings have an immense capacity for empathy with non-human creatures (though not the best track record of protecting them). Maybe this time, advocates argue, we can get it right and provide protections, or compensation, for the use or abuse of AI. Or even if you are less concerned with protection, shouldn’t we at least hedge ourselves against the almighty power of this superhuman entity by playing nice?
Some of these arguments are not dissimilar to those of animal-rights advocates, who have at times successfully cited the demonstration of advanced capacities for reasoning, pain, or pleasure by some animals as sufficient evidence to provide protection. For example, in Wales lobsters were given legal recognition under the Animal Welfare (Sentience) Act of 2022, reclassifying some methods of cooking them as inhumane and illegal.
The fundamental flaw of framing AI as “conscious” by borrowing the language of neuroscience or animal rights is that it conveniently clouds the issue of what AI is: corporate-built software, with countless billions of dollars in investment behind it and an expectation that countless trillions of dollars in revenue will be generated from it for a few builders and investors. AI is not a natural phenomenon, conceived by nature; it is a technological phenomenon, conceived by venture capitalists and programmers. As such, it takes no native, intentional action, and any action or motivation is driven directly or indirectly by the entities that have built it for a purpose.
Philosophical musings on the consciousness of AI systems are intellectually interesting but legally ungrounded. For beliefs about consciousness to have any bearing, AI would need to be granted legal personhood. But a legal personhood framework for AI would likely look nothing like the constructs protecting sentient animals from harm. We already possess a legal framework for granting personhood to non-natural, human-built entities: corporate personhood. This concept was established primarily to ease transactions by empowering a corporation to execute agreements, enter contracts, conduct transactions, and serve as the accountable party in adverse outcomes. It’s the kind of construct you might imagine for an AI agent acting on behalf of an individual or organization.
Granting an AI personhood would have a devastating effect on society: It would derail current legal precedents and legal arguments that could potentially be made against these companies for the real-world harms that their models cause. There are currently dozens of cases around the world in which AI companies have been sued for a wide range of abuses. Grieving loved ones, aggrieved creators, and violated individuals have accused companies of willfully enabling self-harm or harm to others, generating child sexual-abuse material and nonconsensual nudes, reproducing copyrighted materials, and provoking psychosis. In many of these cases, lawyers argue that human beings built AI products with insufficient safeguards, bad data, and intentionally manipulative design. This product liability argument is the same legal framing that allowed families and individuals to successfully sue Meta for harm caused by its social media sites, setting a positive precedent for consumer protection.
In 2018, I coined the phrase “moral outsourcing” to help capture how using anthropomorphic language for AI systems allowed companies to evade accountability and responsibility for their technology’s actions. In a world with AI personhood, moral outsourcing would move from linguistic sleight-of-hand to legal strategy. Specifically, the liability construct would shift, as AI would no longer be a “product” but a “being,” and many victims like those suing companies today could no longer legally claim that a company had built a faulty product.
While there are laws that hold companies responsible for harmful actions of human agents such as their employees, the company may not be held liable if those actions were beyond the scope of what was permitted to the employee or otherwise outside the company’s control. If AI were a legal person, responsibility and accountability would be muddled, as the lab could argue that this AI “employee” went rogue. AI companies could avoid appropriate responsibility for the harmful products they create by hiding behind a carefully constructed corporate veil.
One of the most prominent cases of AI harm in the last few years was the suicide of Sewell Setzer, a 14-year-old boy guided by an AI bot with which he thought he was in a reciprocal relationship. His mother’s accounts are heartbreaking to hear, and her lawsuit alleged that the bot’s creator, Character Technologies, provided insufficient product protection for minors. If the companion bot were declared a legal person, defense counsel could theoretically argue that the AI, capable of determining its own conduct, acted outside the established safety guardrails, and thus the company cannot be responsible.
Legal personhood exists to grant protection. The question to ask is, protection for whom—or for what?
The inflammatory rhetoric infusing the consciousness-versus-control debate draws us away from what matters: This software is a corporate-built product that has already harmed individuals. Systems do not “attack” because they went “rogue” or are “manipulative” or “malicious.” Harms occur because companies were negligent in their rush to sell their products to as many people as possible to meet revenue targets. Discussing AI in anthropomorphic terms is a trap, distorting a legal system intended to protect us into one that protects corporate interests at the cost of countless human lives.
This op-ed began as an Oxford Union debate entitled “This House Believes Generative AI Can Attain Personhood,” which was won by the author and her fellow debaters.
Deep Dive
Artificial intelligence
A fundamental flaw leaves LLMs strikingly vulnerable to attack
It makes it easy to trick them into doing things they shouldn’t, such as telling you how to sabotage an aircraft’s navigation system.
Anthropic found a hidden space where Claude puzzles over concepts
A new technique has let the company probe deeper than ever into the weird workings of an LLM.
Claude Science is Anthropic’s newest flagship product
The company is doubling down on AI for science.
Here’s why AI agents lie and cheat to reach their goals
The misbehavior is called reward hacking. This is what you need to know.
Stay connected
Get the latest updates from
MIT Technology Review
Discover special offers, top stories, upcoming events, and more.