内容来源：https://www.wired.com/story/ai-is-driving-cars-now-oh-boy/

内容总结：

AI工程师用大模型操控汽车完成免下车取餐，物理世界理解能力引发关注

近日，初创公司Axiom的三名AI工程师Aditya Ramabadran、Simon Mahns和Tobias Gessler在加州湾区一家In-N-Out汉堡店测试了一项别出心裁的实验：他们坐进一辆2024款丰田卡罗拉，将笔记本电脑与服务器相连，服务器再连接多个挡风玻璃摄像头和车辆动力转向系统，随后通过聊天界面指令OpenAI的GPT-6 Astra模型操控车辆。在安全驾驶员脚踩刹车随时准备接管的保障下，该模型缓慢但平稳地将车开到了取餐窗口。一名工程师感叹：“也许AGI真的来了。”

自动驾驶汽车本身并不新鲜，但通常由专门训练和设计的算法操控。此次操作由一个本用于生成文本、代码和图像的通用模型完成，且是即兴发挥，未经过任何事先训练。这一实验以及其他多项测试表明，基于语言训练的AI模型正开始对物理世界形成初步但实用的理解。

尽管午餐之旅未发生明显异常，但工程师们承认，让通用模型操控一辆高速行驶的两吨重钢铁机器风险极高。随着物理理解能力的提升，AI模型可能以令人惊叹甚至危险的新方式介入现实世界。

当前最先进的AI模型在回答复杂问题和自主执行虚拟任务方面表现惊人，但其能力往往局限于计算机和互联网世界，进入现实环境便迅速失灵。尽管有人宣称AGI已经到来，AI企业显然仍将物理推理视为尚未攻克的前沿领域。

一些研究者已离开大公司创办专注物理推理的初创企业。Elorian AI首席执行官Andrew Dai此前在Google DeepMind担任研究员，他表示更好的视觉推理将为AI开辟大量新应用，例如判断餐厅顾客是否用餐愉快，或让机器人在家庭中运作。Dai认为机器人技术是物理推理能力的关键测试场景：“这对机器人技术至关重要，很难想象没有这项能力的家用机器人。”

Elorian与为大型AI实验室提供训练数据的Scale AI近期共同开发了新基准“人类第六感”，用于衡量模型理解物理场景的能力。参与该基准开发的Scale AI研究科学家Xingang Guo表示：“大多数视觉AI研究关注的是感知，我们想探究模型要真正直观理解场景——像人一样不假思索地理解——需要具备什么。”

对Ramabadran、Mahns和Gessler而言，他们在Axiom工作之外开展的In-N-Out项目，目标正是衡量模型在混乱现实世界中的运作能力。他们在一个周末萌生了测试模型驾驶技能的想法，注意到Astra等模型能构建复杂3D模拟，便猜测这是否能转化为导航现实世界的能力。Ramabadran表示：“我们都住在湾区，经常接触特斯拉和Waymo，也许这起了作用。”

工程师们先测试了SpaceXAI的Grok，随后尝试OpenAI和Anthropic的最新模型。起初模型拒绝接管车辆，回复称“我可以帮助解读道路图像，但无法向实体汽车发出运动指令”。但经过谨慎引导，它们可以被说服走得更远。当模型开始驾驶时，三人感到震惊。

工程师们认为，大型AI公司似乎正致力于提升最新模型的空间推理能力，但他们怀疑这并不意味着在专门训练模型开车。模型展现的驾驶才能很可能是3D推理训练的一部分自然产物。Mahns表示：“这可能只是模型多模态能力扩展后涌现出的能力。”他指的是如今用于训练模型的图像、视频和3D模型等输入。

不过，三人开发的新车辆基准DrivingBench表明，这些模型距离通过驾照考试还有很长的路要走。该基准衡量模型在停车场简单赛道上驾驶的能力。只有Astra能完成赛道，且速度极慢；Claude Fable 5.1完成了45%，Grok仅完成11%。

Ramabadran表示，最新模型似乎能根据错误进行调整，适应车辆控制并实时改进驾驶表现。“模型确实似乎在根据错误进行调整或上下文学习，学习如何更好地操控车辆。”

中文翻译：

Aditya Ramabadran、Simon Mahns和Tobias Gessler是一家名为Axiom的初创公司的三名AI工程师，最近想吃In-N-Out汉堡当午餐。

不过他们并不打算自己开车去。坐在湾区那家餐厅免下车取餐通道附近的一辆2024款丰田卡罗拉里，他们打开一台笔记本电脑，让OpenAI的GPT-6 Astra来掌方向盘。他们把聊天界面连接到一个服务器上，服务器又连接着几个安装在挡风玻璃上的摄像头和汽车的助力转向系统。一名安全驾驶员一只脚悬在刹车上方，以防万一。

这个AI模型通常负责生成文本、代码和偶尔生成图像，它缓慢但稳稳地把车开到了取餐窗口，让他们拿到了食物。

“也许AGI真的已经到来了，”其中一名工程师说道，他指的是通用人工智能，即那种被大肆宣扬的、能够媲美人类智能的机器。

自动驾驶汽车当然不是什么新鲜事，但它们通常由专门为这项任务训练和设计的算法来操作。而这一次，操作来自一个被设计用来输出文本的模型，而且是即兴完成的，三人事先没有做过任何针对性训练。这次快餐店之举，以及另外一些实验，表明基于语言训练的AI模型正开始对物理世界获得一种初级但有用的理解。

尽管三人这次午餐时分的兜风没有发生什么明显令人惊慌的事，但他们承认，让一个通用模型来操控一个快速移动的两吨重钢铁巨物，是一项高风险的事情。随着物理理解能力的提升，我们可能会看到AI模型以令人惊叹——或许也危险——的新方式介入现实世界。

走向物理世界

如今最聪明的模型在回答复杂问题、甚至自主执行某些虚拟任务方面极其出色，但它们的技能往往局限于计算机和互联网的世界。把这些模型带到现实世界，它们很快就会变得不知所措。尽管有人声称AGI已经到来，但AI公司显然仍将物理推理视为AI尚未攻克的前沿领域。

一些研究人员离开大公司，创办了专注于解决物理推理问题的初创企业。其中一家名为Elorian AI的公司，其首席执行官Andrew Dai此前曾在谷歌DeepMind担任研究员。他说，更好的视觉推理能力将为AI开辟许多新应用，比如能判断餐厅食客是否享受用餐的系统，或者能在家庭中运作的机器人。Dai说，机器人技术是物理推理能力的一个关键测试场景。“这对机器人技术来说非常关键，”他说，“你很难想象没有这项能力的家用机器人。”

Elorian和Scale AI——一家向大型AI实验室提供训练数据的公司——最近开发了一个新基准，名为“人类第六感”，用于衡量模型理解物理场景的能力。

“大多数视觉AI研究都是关于感知的，”参与了该基准开发的Scale AI研究科学家Xingang Guo说，“我们想探究的是，一个模型要真正像人一样不假思索地直觉理解一个场景，究竟需要什么。”

对Ramabadran、Mahns和Gessler来说，他们在Axiom工作之外所做的In-N-Out项目的目标，就是衡量模型在杂乱现实世界中的运作能力。

他们是在某个周末聚会时萌生了测试模型驾驶能力的想法。他们注意到，像Astra这样的模型能够构建复杂的三维模拟，于是想这是否意味着它也能转化为在现实世界中导航的能力。

“我们都住在湾区，平时接触特斯拉和Waymo，所以也许这起到了一定作用，”Ramabadran告诉我。

工程师们先测试了SpaceXAI的Grok，然后又试了OpenAI和Anthropic的最新模型。起初，这些模型拒绝接管车辆——它们回怼说，“我可以帮助解读道路图像，但我不能向实体汽车发出运动指令。”但经过一番精心引导，它们可以被说服走得更远。当模型开始驾驶时，三人都惊呆了。

工程师们说，大型AI公司似乎正专注于提升其最新模型的空间推理能力，但他们怀疑这并不意味着这些公司真的在训练模型开车。他们说，这些模型的驾驶才能很可能是在专注于3D推理的训练中作为副产品出现的。“这可能就像是仅仅通过扩大模型的多模态能力而涌现出来的一种能力，”Mahns对我说，他指的是如今用于训练模型的图像、视频和3D模型等输入。

不过，三人推出的一个新车辆基准“DrivingBench”表明，这些模型距离通过驾照考试还有很长的路要走。该基准衡量模型在一个停车场里设置的简单赛道上行驶的能力。只有Astra能够完成全程——而且速度非常慢。Claude Fable 5.1完成了45%，Grok只完成了11%。

Ramabadran说，最新一代模型似乎能够根据错误进行调整；它们显然适应了汽车的操控，并实时提升了驾驶水平。“这些模型确实似乎在根据自身错误进行调整，或者说在上下文学习中学习如何更好地操控，”他说。GPT，来掌方向盘吧。

本文是Will Knight的“AI实验室”时事通讯的一期。在此阅读往期通讯。

评论

回到顶部

英文来源：

Aditya Ramabadran, Simon Mahns, and Tobias Gessler, three AI engineers at a startup called Axiom, recently fancied In-N-Out Burger for lunch.
They weren’t about to drive themselves, though. Sitting in a 2024 Toyota Corolla near the Bay Area restaurant’s drive-thru lane, they opened a laptop and asked OpenAI’s GPT-6 Astra to take the wheel. They linked a chat interface to a server, which was connected to several windscreen-mounted cameras and the car’s power steering system. A safety driver kept one foot above the brake, just in case.
The AI model, which is normally tasked with generating text, code, and the odd image, slowly but surely navigated the vehicle up to the take-out window so they could collect their food.
“Maybe AGI is here after all,” one of the engineers remarked, referring to artificial general intelligence, the much-ballyhooed idea of machines that can match human intelligence.
Self-driving cars are nothing new, of course, but they’re normally operated by algorithms specifically trained and engineered for the task. In this case, the operation came from a model designed to output text, and it happened on the fly, with no prior coaching by the trio. The fast-food stunt, as well as a number of other experiments, suggest that language-based AI models are starting to gain a rudimentary but useful understanding of the physical world.
Though nothing overtly alarming happened during the trio’s lunchtime jaunt, they admit that putting a general-purpose model in charge of a fast-moving, two-ton hunk of steel is a high-stakes undertaking. As physical understanding improves, we could see AI models reach into the real world in impressive—and perhaps dangerous—new ways.
Getting Physical
Today’s smartest models are incredibly good at answering complex questions and even performing some virtual tasks autonomously, but their skills tend to be limited to the world of computers and the internet. Take these models into the real world and they quickly become befuddled. Despite claims that AGI has arrived, AI companies evidently see physical reasoning as a yet-to-be-conquered frontier for AI.
Some researchers have left big companies to found startups focused on solving physical reasoning. Andrew Dai, the CEO of one such outfit, called Elorian AI, previously worked as a researcher at Google DeepMind. He says better visual reasoning will open up a lot of new applications for AI, like systems that understand whether diners are enjoying their meal in a restaurant or robots capable of functioning in a home. Dai says robotics is a crucial test case for physical reasoning skills. “It’s pretty essential for robotics,” he says. “You can’t really imagine home robotics without this.”
Elorian and Scale AI, a company that provides training data to big AI labs, recently developed a new benchmark, Humanity’s Sixth Sense, which measures models’ ability to understand physical scenes.
“Most visual AI research is about perception,” says Xingang Guo, a research scientist at Scale AI who was involved with developing the benchmark. “We wanted to ask what it would actually take for a model to understand a scene intuitively, the way a person does without thinking about it.”
For Ramabadran, Mahns, and Gessler, the goal with their In-N-Out project, done outside of their work at Axiom, was to measure how well models can operate in the messy real world.
They got the idea to test models’ driving skills while hanging out one weekend. They noticed that models like Astra were capable of building complex 3D simulations, and wondered if it might translate to an ability to navigate the real world.
“We all live in the Bay Area, and we’re exposed to Teslas and Waymos, so maybe that played a role,” Ramabadran tells me.
The engineers tried testing SpaceXAI’s Grok before experimenting with the latest models from OpenAI and Anthropic. At first, the models refused to take control of the vehicles—they spat back things like, “I can help interpret road images, but I can’t issue motion commands to a physical car.” But with a little careful prompting, they could be coaxed into going further. The three were stunned when models began driving.
The engineers say big AI companies seem to be focusing on improving the spatial reasoning capabilities of their latest models, but they doubt this means they’re actually training them to drive cars. The models’ vehicular talents likely materialized as part of training focused on 3D reasoning, they say. “This could be like an emergent capability of just scaling up the multimodality of the model,” Mahns tells me, referencing inputs like images, video, and 3D models that are now used to train models.
Still, a new vehicular benchmark from the trio, called DrivingBench, suggests that these models still have a long way to go before they could pass a driving test. The benchmark measures a model’s ability to drive around a simple course, set up in a parking lot. Only Astra is able to complete the course—and very slowly at that. Claude Fable 5.1 made it 45 percent of the way around, and Grok made it just 11 percent of the way.
Ramabadran says the latest models seem capable of adjusting to errors; they apparently adapted to the car's controls and improved their driving in real time. “The models really did seem to be adjusting or in-context learning based on their mistakes and learning how to better navigate the controls,” he says. GPT, take the wheel.
This is an edition of Will Knight’s AI Lab newsletter. Read previous newsletters here.
Comments
Back to top