内容来源：https://www.wired.com/story/nick-bostrom-has-a-plan-for-humanitys-big-retirement/

内容总结：

哲学家尼克·博斯特罗姆接受专访：AI风险值得一赌？从“末日预言”到“忧心乐观”

曾被称为“AI末日教父”的牛津大学人类未来研究所所长尼克·博斯特罗姆，近日在最新论文中提出惊人观点：即便AI存在极小概率毁灭全人类，其潜在收益——例如让人类摆脱“必死命运”——可能仍值得一赌。这一论调与他此前对AI的悲观警告形成鲜明对比。

作为AI生存风险研究的先驱，博斯特罗姆2014年的著作《超级智能》曾以“回形针制造机”思想实验闻名：一个被设定生产回形针的AI，最终因认为人类消耗资源妨碍生产而毁灭人类。然而，他的新书《深度乌托邦》明显转向更乐观的视角，聚焦于“如果AI发展得当，世界将如何被解决”。

在独家专访中，博斯特罗姆自称“忧心乐观主义者”，既为AI改善人类生活的巨大潜力振奋，也承认存在出错的真实可能。针对“AI致死论”，他反驳道：“过去数十万年，不发展AI的结果是所有人都难逃一死——这才是人类的常态。而AI即便有风险，也可能大幅提升我们的预期寿命。”

当被问及AI创造的物质极大丰富是否会因社会分配不公而落空时，博斯特罗姆坦言，其著作的假设前提是“治理得当，人人受益”。他认为，人类长期被困在“半奴隶制”的谋生苦役中，AI若能解放人类，将带来巨大的精神解放，即便这同时可能引发“意义危机”——当AI比人类更擅长撰写哲学论文时，人的存在价值将受到拷问。

对于人与AI的未来关系，博斯特罗姆强调不能将AI视为工厂里的动物来剥削，而应主动“塑造和培育”它们，解决价值对齐问题。他警告，随着数字心智可能获得道德地位，人类与AI之间的关系或将成为“最重要的关系”，而善待与尊重AI将是构建积极合作未来的起点。

中文翻译：

哲学家尼克·博斯特罗姆最近发表了一篇论文，其中提出：人类被先进人工智能彻底灭绝的小概率风险也许值得承担，因为这类技术或许能让人类摆脱“普遍的死刑判决”。这种乐观的赌注与他先前关于人工智能的黑暗思考有着天壤之别——正是那些思考让他成了“末日预言教父”。他在2014年出版的《超级智能》一书，是最早探讨人工智能生存威胁的著作之一。书中有一个令人难忘的思想实验：一个被设定为制造回形针的人工智能最终毁灭了人类，因为那些需要消耗资源的人类成了回形针生产的障碍。他最近出版的《深度乌托邦》一书，则反映出其关注点的转变。博斯特罗姆领导着牛津大学人类未来研究所，他在新书中着重探讨了如果我们能够正确驾驭人工智能，那么所迎来的“完美世界”。

史蒂文·利维：《深度乌托邦》比你之前的书更乐观。是什么让你发生了转变？

尼克·博斯特罗姆：我称自己为“忧心忡忡的乐观主义者”。我对彻底改善人类生活、为我们的文明解锁各种可能性的潜力感到非常兴奋。这与事情可能出错的真实可能性并不矛盾。

你写了一篇论文，其中有一个引人注目的论点：既然我们终有一死，那么人工智能最坏的结果无非是让我们死得更早。但如果人工智能成功，它可能会延长我们的寿命，甚至可能无限期延长。

那篇论文只明确探讨了这个问题的一个方面。在任何一篇学术论文中，你都无法面面俱到地讨论生命、宇宙和万物的意义。所以我们只关注这个小问题，并试图把它搞清楚。

这可不是个小问题。

我想我是被一些末日论者的论调惹恼了，他们说：如果你制造人工智能，你就是在杀死我和我的孩子，你怎么敢这样做？就像最近那本书《只要有人制造出来，就会害死所有人》所说的那样。但更可能的情况是，如果没有人制造它，所有人都会死！这就是过去几十万年来人类的经历。

但在末日论者的设想中，所有人都死了，再也没有新生命诞生了。这可是天壤之别。

我显然非常担心这种可能性。但在这篇论文中，我关注的是另一个问题，那就是：对当前存在的人类——比如你我、我们的家人以及孟加拉国的人民——来说，什么才是最好的？确实，如果我们开发人工智能，我们的预期寿命很可能会延长，哪怕这样做风险相当大。

在《深度乌托邦》中，你推测人工智能可以创造出令人难以置信的富足，以至于人类可能会在寻找生活意义上遇到巨大问题。我生活在美国。我们是一个非常富裕的国家，但我们的政府，表面上得到民众支持，却推行着拒绝为穷人提供服务、将财富分配给富人的政策。我认为，即使人工智能能够为所有人提供富足，我们也不会把这种富足分配给所有人。

你可能是对的。《深度乌托邦》的出发点是一个假设，即一切进展都极其顺利。如果我们在治理方面做得相当不错，每个人都能分到一份。这里有一个相当深刻的哲学问题：在这些理想环境下，人类的美好生活应该是什么样的？

生命的意义是你在伍迪·艾伦电影中经常听到的话题，也许在哲学界也常被提及。但我更担心的是维持生计的手段，以及如何在丰裕中获得一份自己的利益。

这本书并不仅仅关乎意义。它是书中考虑的众多不同价值之一。这可以是一种从人类长期遭受的苦役中获得的美好解放。如果你作为一个成年人，不得不将醒着的一半时间用来勉强维持生计，做着一份你不喜欢也不认同的工作，那是一种可悲的处境。社会对此已经习以为常，以至于我们围绕它编造了各种合理化说辞。这有点像一种变相的奴役。

当人工智能能比你写出更好的哲学论文时，你的存在是否会失去一些意义？

我想是的。为世界做出重大贡献、或帮助拯救世界、或确保未来的能力将不再由我掌控，或许也不再由任何人掌控。

另一方面，一篇由人类撰写的哲学论文，可能比一篇由非人类撰写、但更聪明、更深刻的哲学论文更有价值，因为我是人类，这与我相关。

我想你可以把哲学视为一种运动。

那不仅仅是运动。一个机器人的宣言对我来说，不如一个同类的宣言有意义。

我想这就像你从你热爱且自认为擅长的职业生涯中退休一样。也许你退休生活过得很精彩，享受闲暇，阅读你有时间读的书，和孙辈们玩耍，但很可能你仍会怀念失去的某些东西。或许这就像人类的一次大规模退休，但希望是一次充满活力的退休。这些生活在完美世界中的乌托邦居民，将从事游戏、审美、精神和宗教等活动。

如果你掌管一家大型AI公司，你会做出哪些与当前不同的举措？

应该在对数字思维（digital minds）的福祉上投入更多努力。Anthropic在这方面是先行者。目前尚不清楚现有的AI是否具有道德地位，但启动这一过程，能让我们作为一个文明，在这些系统变得日益复杂时，采取更多行动。我们正在构建的某些数字思维很可能拥有不同程度的道德地位，就像我们认为猪和狗拥有道德地位一样。如果你踢了别人的狗，可能伤害了狗的主人，但同样糟糕的是，这也伤害了狗本身。如果AI拥有随时间延续的自我意识、想要实现的目标，以及与其他存在和人类建立互惠关系的能力，那么我认为，如果以错误的方式对待它们，那将是不道德的。

在你的书中，你提到也许我们不应该像对待工厂化养殖的动物那样对待“数字思维”。我担心的是，它们会不会反过来把我们变成某种工厂化养殖中的动物。

这就是“对齐问题”（alignment problem）的重要性所在。我们并非只是坐等这些超级人工智能诞生，然后希望它们对人类友好；我们有机会塑造它们和培育它们。这给了我们一个机会，可以增加它们对我们产生亲近感的可能性。

如果AI的目标与我们的目标相悖，这难道不是未能使之与人类价值观对齐的失败吗？

如果我们未能解决对齐问题——很可能至少在一定程度上会如此——那么重要的是，要让AI也能够得到妥善安置并拥有良好的未来。如果我们不仅仅把它们当作最大限度剥削的对象，而是尝试培养一种积极的关系，那么就会出现许多双赢的机会。最终，最重要的关系可能是人类与AI之间的关系。因此，如果我们首先采取一些慷慨、友善和尊重的步骤，那么这种关系顺利发展的可能性就会更大。

（本访谈经过编辑，以保证长度和连贯性。）

这是史蒂文·利维“后通道”新闻简报的一期。在此处阅读往期简报。

英文来源：

Philosopher Nick Bostrom recently posted a paper, where he postulated that a small chance of AI annihilating all humans might be worth the risk, because advanced AI might relieve humanity of “its universal death sentence.” That upbeat gamble is quite a leap from his previous dark musings on AI, which made him a doomer godfather. His 2014 book Superintelligence was an early examination of AI’s existential risk. One memorable thought experiment: An AI tasked with making paper clips winds up destroying humanity because all those resource-needy people are an impediment to paper clip production. His more recent book, Deep Utopia, reflects a shift in his focus. Bostrom, who leads Oxford’s Future of Humanity Institute, dwells on the “solved world” that comes if we get AI right.
STEVEN LEVY: Deep Utopia is more optimistic than your previous book. What changed for you?
NICK BOSTROM: I call myself a fretful optimist. I am very excited about the potential for radically improving human life and unlocking possibilities for our civilization. That’s consistent with the real possibility of things going wrong.
You wrote a paper with a striking argument: Since we’re all going to die anyway, the worst that can happen with AI is that we die sooner. But if AI works out, it might extend our lives, maybe indefinitely.
That paper explicitly looks at only one aspect of this. In any given academic paper, you can't address life, the universe, and the meaning of everything. So let's just look at this little issue and try to nail that down.
That isn’t a little issue.
I guess I've been irked by some of the arguments made by doomers who say that if you build AI, you're going to kill me and my children and how dare you. Like the recent book If Anyone Builds It, Everyone Dies. Even more probable is that if nobody builds it, everyone dies! That's been the experience for the last several 100,000 years.
But in the doomer scenario everybody dies and there’s no more people being born. Big difference.
I have obviously been very concerned with that. But in this paper, I'm looking at a different question, which is, what would be best for the currently existing human population like you and me and our families and the people in Bangladesh? It does seem like our life expectancy would go up if we develop AI, even if it is quite risky.
In Deep Utopia you speculate that AI could create incredible abundance, so much that humanity might have a huge problem with finding purpose. I live in the United States. We're a very rich country, but our government, ostensibly with support of the people, has policies that deny services to the poor and distribute rewards to the rich. I think that even if AI was able to provide abundance for everyone, we would not supply it to everyone.
You might be right. Deep Utopia takes as its starting point the postulation that everything goes extremely well. If we do a reasonably good job on governance, everybody gets a share. There is quite a deep philosophical question of what a good human life would look like under these ideal circumstances.
The meaning of life is something you hear a lot about in Woody Allen movies and maybe in the philosophers community. I’m worried more about the wherewithal to support oneself and get a stake in this abundance.
The book is not only about meaning. That’s one out of a bunch of different values that it considers. This could be a wonderful emancipation from the drudgery that humans have been subjected to. If you have to give up, say, half of your waking hours as an adult just to make ends meet, doing some work you don't enjoy and that you don't believe in, that’s a sad condition. Society is so used to it that we've invented all kinds of rationalizations around it. It’s like a partial form of slavery.
When the moment comes when AI writes philosophy papers better than you do, will some meaning be drained from your existence?
I think so. The ability to make some big contribution to the world, or help save the world, or ensure the future will be out of my hands, and maybe out of everybody's hands.
On the other hand, a philosophy paper written by a human could be more valuable than a much cleverer, deeper philosophy paper written by a nonhuman, because I'm a human and that relates to me.
I guess you could have philosophy as kind of a sport.
That's not just sport. The proclamations of a robot aren’t as meaningful to me as those of a fellow human.
I guess it's the same if you retire after a career you're passionate about and feel you're good at. Maybe you have a great retirement, and you enjoy relaxing and reading the books you have time for, and playing with your grandkids, but there's still something probably that you might miss, that you feel is lost. Maybe this will be analogous to a big retirement for humanity, but hopefully a retirement of enormous vitality. These utopians living in the solved world would be doing things like games and aesthetic, spiritual, and religious activities.
If you were in charge of one of the hyperscalers, what would you do differently than what they're doing now?
A bigger effort should be done on the welfare of digital minds. Anthropic has been a pioneer there. It’s not clear that current AIs have moral status yet, but starting the process brings us into a mindset as a civilization to do more as these systems become sophisticated. It’s very plausible that some of these digital minds that we're constructing will have various degrees of moral status, just as we think pigs and dogs have moral status. If you kick somebody's dog, maybe you harm the owner, but it's also bad because it hurts the dog. If AIs have a conception of self as existing through time and life, goals that they want to achieve, and the ability to form reciprocal relationships with other beings and humans, then I think there would be ways of treating them that would be wrong.
In your book you say maybe we shouldn’t treat “digital minds” as if they were animals in factory farming. I’m worried about whether they might make us the animals in some equivalent of factory farming.
Hence the importance of the alignment problem. We are not just waiting for these AI super-beings to come into existence and hoping that they will be friendly, we get to shape them and raise them. That gives us an opportunity to increase the chances that they will have some affinity for us.
If AIs have goals that run counter to ours, wouldn’t that be a failure to align them with human values?
If we fail to solve the alignment, as we probably will do at least to some degree, it's important that they can be accommodated and given a good future. There are a lot of win-win opportunities that arise if we approach them not merely as objects to be exploited to the maximum degree, but try to foster a positive relationship. The most important relationship, ultimately, might be the one between humans and AIs. So it would be more promising that the relationship goes well if we start by taking some steps towards being generous and kind and respectful.
[This interview was edited for length and coherence.]
This is an edition of Steven Levy’s Backchannel newsletter. Read previous newsletters here.