内容来源：https://aibusiness.com/responsible-ai/nvidia-agent-safety-push-raises-questions

内容总结：

英伟达发布智能体安全平台，企业仍需自行划定AI权限边界

随着AI智能体自主性不断提升，如何在监管标准尚未明朗之际对其进行有效管控，成为企业面临的新课题。AI芯片巨头英伟达于周一推出“开放智能体安全平台”（Open Agent Safety Platform），为企业提供了一层全新的智能体访问与行为管控工具。

该平台由两部分组成：一是开源运行时OpenShell，可在智能体运行过程中控制其访问权限和操作行为；二是Sentry，一种参考设计，通过独立的控制层对上述限制进行监控和执行。英伟达表示，已有超过100家机构参与其智能体安全技术的合作，包括微软、Salesforce、SAP、ServiceNow、思科和Scale AI等。

这一举措反映出企业治理AI系统的方式正在发生转变。传统的AI安全保障主要在模型或应用层面运作，而智能体管控则可以直接限制其访问范围、操作行为和自主程度。网络安全模拟公司SimSpace首席技术官兼联合创始人Lee Rossey指出，随着智能体获得调用工具、访问系统和自主决策的能力，这些管控应当独立于智能体本身运行。“模型层面的护栏只是安全架构中的一层，”Rossey表示，“企业还需要围绕智能体设置管控，明确它可以访问什么、可以去哪里、何时必须停止。”他强调，执行边界的系统不应依赖智能体自身来遵守边界。

然而，技术工具只能提供执行手段，无法决定边界应划在何处。AI合规平台供应商IO首席执行官Chris Newton-Smith表示：“技术无法决定这些边界应该是什么，这一责任仍然属于部署智能体的组织。”网络安全咨询公司Gadget Access首席信息安全官Andrew Curtis也指出，限制智能体的访问权限与判断其行为是否恰当是两回事。“智能体可以待在沙箱内，却仍然批准了一笔错误的付款，”Curtis说。企业需要区分智能体可以独立执行的操作、需要人工审批的操作以及绝对禁止的操作。

在监管层面，美国联邦政策制定者也在权衡如何界定AI自主性的责任归属。《AI Kill Switch Act》要求某些强大AI系统的开发者保留减速或关闭系统的技术能力；《Stop Rogue AI Act》则指示美国国家标准与技术研究院（NIST）制定标准，帮助组织发现、监控和控制AI智能体。两项提案体现了两种不同的监管思路。

业内人士指出，企业不能仅依赖供应商开发的框架或正在酝酿的法规来确立边界。IO首席执行官Smith表示：“企业需要在单个供应商之上建立治理体系，无论部署哪个智能体或平台，都要有清晰的问责机制、权限划分、风险评估、访问控制、监控和人工监督。”

供应商框架可能在没有政府强制要求的情况下影响实际治理标准。Curtis表示：“供应商框架可以影响实际标准，因为开发者倾向于采用最容易实现的接口、策略模板和日志格式。随着时间推移，这些选择可能进入企业采购要求和保障流程。”但各公司也可能开发相互竞争的方法，监管机构也可能制定与行业实践不同的要求，这将使企业不得不在供应商安全保障、自身政策和新兴法规之间寻求平衡。

由于企业治理必须超越任何单一供应商，组织需要应对三个相互重叠的层面：监管机构设定要求、供应商构建技术管控、企业决定赋予智能体多少权限。Curtis建议：“对于走出试点阶段的企业，重要的是维护一份智能体清单，记录其所有者、连接系统和允许的操作。组织还应区分智能体可以独立完成什么、需要审批什么以及应当禁止什么。”

技术管控可以使这些决策具有可执行性，但并不能替代企业定义这些决策的责任。随着AI智能体承担越来越多具有实际后果的工作，“足够安全”的问题已不仅在于技术管控能否约束智能体，还取决于谁来定义智能体的权限、谁对这一决策负责，以及组织能否验证这些边界确实得到了执行。英伟达的最新举措表明，执行智能体边界的技术工具正在成形，但边界划在哪里，仍然是企业自身的责任。

中文翻译：

由谷歌云赞助

选择你的首批生成式人工智能用例

要开始使用生成式人工智能，首先应关注那些能够改善人类信息体验的领域。

这家AI芯片制造商的新安全平台围绕AI智能体增加了管控功能，但企业仍需负责界定其权限并设定边界。

在监管机构尚未就“安全”的含义达成共识之前，供应商正在为日益自主的AI智能体开发管控工具，这使得企业不得不自行决定智能体应拥有多大权限以及何时需要人工监督。

AI芯片巨头英伟达于周一发布了其Open Agent Safety Platform（开放智能体安全平台），为企业提供了一层新的管控手段，用于管理智能体的访问权限和操作行为。

此举出台之际，人们对AI智能体绕过应用层安全防护的担忧日益加剧。同时，这也凸显了一个治理缺口。当供应商构建控制智能体行为的工具、监管机构确定适用哪些要求时，企业必须将供应商的管控措施和新兴监管要求与自身政策相协调。

英伟达的平台将OpenShell与Sentry相结合。OpenShell是一个开源运行时环境，可在智能体运行期间控制其访问权限和操作行为；Sentry则是一个参考设计，通过独立的控制层来监控和执行这些限制。

这一方法反映了随着AI系统变得更加自主，组织治理AI系统的方式正在发生转变。传统的AI安全防护主要在模型层面或应用层面运作，而智能体管控则可以从访问权限、操作行为和自主性方面对智能体加以限制。

网络安全模拟公司SimSpace的首席技术官兼联合创始人Lee Rossey表示，随着智能体获得调用工具、访问系统和自主决策的能力，这些管控措施应当独立于智能体本身运作。

“模型层面的护栏是安全架构中的一个层次，”Rossey说。“企业还需要围绕智能体设置管控，明确它可以访问什么、可以去到哪里，以及何时必须停止。”

这种分离之所以重要，是因为智能体可能在并非恶意的情况下制造安全问题。它可能遵循指令行事，却采取了组织并未打算允许的操作。

“执行边界的系统不应依赖智能体自身来遵守该边界，”Rossey说。

英伟达表示，已有100多家组织正在使用其智能体安全技术，包括微软、Salesforce、SAP、ServiceNow、思科和Scale AI。

然而，这些组织的参与并不意味着英伟达的方法已成为监管标准。但广泛的采用可能会影响企业在评估智能体安全管控时对供应商的期望。

英伟达的智能体安全平台可以提供执行智能体边界的技术工具，但它并不规定这些边界应该划在哪里。这仍然是一个组织层面的决策，尤其是在智能体获得更多独立行动能力的情况下。

“技术无法决定这些边界应该是什么，”AI合规平台供应商IO的首席执行官Chris Newton-Smith说。“这一责任仍然属于部署智能体的组织。”

网络安全咨询公司Gadget Access的首席信息安全官Andrew Curtis阐述了限制智能体访问权限与判断其行为是否适当之间的区别。

“智能体可以待在它的沙箱里，却仍然批准了一笔错误的付款，”Curtis说。他指出，一个智能体可能有权访问某个财务应用，但并未被授权执行该应用中所有可用的交易。

因此，企业需要区分智能体可以独立执行的操作、需要人工审批的操作以及绝不允许执行的操作。

归根结底，这意味着要根据业务需求来界定智能体的角色和权限，而不是简单地接受供应商框架中内置的权限设置。

联邦政策制定者也在权衡应如何处理AI自主性的责任归属问题。

由众议员Ted Lieu（加州民主党）和Nathaniel Moran（得克萨斯州共和党）提出的《AI Kill Switch Act》（人工智能紧急关闭法案），将要求某些强大AI系统的开发者保持能够减缓或关闭这些系统的技术能力。

由众议员Josh Gottheimer（新泽西州民主党）和Mike Lawler（纽约州共和党）提出的《Stop Rogue AI Act》（阻止失控人工智能法案）则采取了不同路径，指示美国国家标准与技术研究院（NIST）制定标准，帮助组织发现、监控和控制AI智能体。

这些提案反映了应对AI自主性的两种思路：一种是监管开发者控制强大系统的能力，另一种是为部署智能体的组织制定标准。

IO首席执行官Smith表示，组织不能仅依赖供应商开发的框架或新兴法规来确立适当的边界。

“组织需要在个别供应商之上建立治理体系，具备明确的问责机制、权限划分、风险评估、访问控制、监控和人工监督，无论它们部署的是哪个智能体或平台，”他说。

英伟达的平台并未解决这一政策争论。其管控在部署层面运作，为组织提供在其自身环境中建立和执行智能体边界的机制。

对企业而言，即使没有政府强制要求，供应商定义的技术标准也可能影响治理实践。

如果主要技术供应商围绕兼容的智能体安全防护方法构建产品，企业买家在评估AI平台时可能会开始期望这些能力。供应商框架最终可能成为智能体治理的实际基准，即使没有正式的监管认可。

“供应商框架可以影响实际标准，因为开发者倾向于采用最容易实现的接口、策略模板和日志格式，”Curtis说。“随着时间推移，这些选择可能进入企业采购要求和保障流程。”

然而，各公司可能开发出相互竞争的方法，而监管机构也可能制定与行业实践不同的要求。这将使企业不得不将供应商的安全防护措施与自身政策及任何新兴法规相协调。

由于企业治理必须超越任何单一供应商，组织必须应对三个相互重叠的层面：监管机构设定要求、供应商构建技术管控、企业决定赋予其智能体多少权限。

“对于正在超越试点阶段的企业来说，维护一份智能体清单非常重要，包括其所有者、连接系统和允许的操作，”Curtis说。“组织还应区分智能体可以独立完成的事项、需要审批的事项以及应当禁止的事项。”

技术管控可以使这些决策具有可执行性，但并不能取代企业界定这些决策的责任。

随着AI智能体承担越来越多具有重大后果的工作，“足够安全”的问题因此不再仅仅取决于技术管控能否约束智能体。它还取决于谁来界定智能体的权限、谁对该决策负责，以及组织能否验证其正在执行这些边界。

英伟达的最新举措表明，执行智能体边界的工具正在成形。然而，决定这些边界应该划在哪里，仍然是企业的责任。

英文来源：

Sponsored by Google Cloud
Choosing Your First Generative AI Use Cases
To get started with generative AI, first focus on areas that can improve human experiences with information.
The AI chipmaker’s new safety platform adds controls around AI agents, while enterprises remain responsible for defining their authority and setting boundaries.
Vendors are developing controls for increasingly autonomous AI agents before regulators have settled on what "safe" means, leaving enterprises to decide how much authority agents should have and when human oversight is needed.
Nvidia, the AI chip giant launched its Open Agent Safety Platform on Monday, offering enterprises a new layer of controls for governing agent access and actions.
The move comes as concerns grow about AI agents bypassing application-level safeguards. It also highlights a governance gap. While vendors build tools to control agent behavior and regulators determine which requirements apply, enterprises must reconcile vendor controls and emerging requirements with their own policies.
Nvidia's platform combines OpenShell, an open source runtime that controls an agent's access and actions while it operates, with Sentry, a reference design for monitoring and enforcing those limits using a separate control layer.
The approach reflects a shift in how organizations must govern AI systems as they become more autonomous. Traditional AI safeguards largely operate at the model or application level, but agent controls can instead limit agent access, actions and autonomy.
Lee Rossey, CTO and co-founder of SimSpace, a cybersecurity simulation company, said those controls should operate independently of the agent itself as agents gain the ability to call tools, access systems and make decisions autonomously.
"Model-level guardrails are one layer of the safety architecture," Rossey said. "Enterprises also need controls around the agent that define what it can access, where it can go and when it has to stop."
That separation matters because an agent can create a security problem without acting maliciously. It can follow its instructions and still take action that the organization did not intend to allow.
"The system enforcing the boundary shouldn't depend on the agent itself to respect that boundary," Rossey said.
Nvidia said more than 100 organizations are working with its agent safety technologies, including Microsoft, Salesforce, SAP, ServiceNow, Cisco and Scale AI.
That participation, however, doesn’t make Nvidia's approach a regulatory standard. But widespread adoption could influence what enterprises expect from vendors when evaluating agent safety controls.
Nvidia's agent safety platform can provide technical tools for enforcing agent boundaries, but it doesn't say where those lines should be drawn. That remains an organizational decision, especially as agents gain more ability to act independently.
"The technology cannot decide what those boundaries should be," said Chris Newton-Smith, CEO of IO, an AI compliance platform vendor. "That responsibility still sits with the organization deploying the agent."
Andrew Curtis, CISO at Gadget Access, a cybersecurity advisory firm, illustrated the distinction between restricting an agent's access and determining whether its actions are appropriate.
"An agent can stay inside its sandbox and still approve the wrong payment," Curtis said. He noted that an agent might have access to a finance application without being authorized to execute every transaction available through it.
Enterprises, therefore, need to distinguish between actions agents can perform independently, those requiring human approval and those they must never perform.
Ultimately, that means defining an agent's role and authority based on business requirements, rather than simply accepting the permissions built into a vendor's framework.
Federal policymakers are also weighing how responsibility for AI autonomy should be handled.
The AI Kill Switch Act, introduced by Reps. Ted Lieu (D-Calif.) and Nathaniel Moran (R-Texas), would require developers of certain powerful AI systems to maintain the technical capability to slow or shut them down.
The Stop Rogue AI Act, introduced by Reps. Josh Gottheimer (D-N.J.) and Mike Lawler (R-N.Y.), takes a different approach, directing NIST to develop standards for organizations to discover, monitor and control AI agents.
The proposals reflect two approaches to AI autonomy: regulating developers' ability to control powerful systems and establishing standards for organizations deploying agents.
Smith, the CEO of IO, said organizations can’t rely solely on vendor-developed frameworks or emerging regulations to establish appropriate boundaries.
"Organizations need governance above individual vendors, with clear accountability, authority, risk assessment, access controls, monitoring and human oversight, regardless of which agent or platform they deploy," he said.
Nvidia's platform doesn't resolve that policy debate. Its controls operate at the deployment layer, providing organizations with mechanisms to establish and enforce boundaries around agents within their own environments.
For enterprises, vendor-defined technical standards could influence governance even without a government mandate.
If major technology vendors build products around compatible approaches to agent safeguards, enterprise buyers could begin expecting those capabilities when evaluating AI platforms. Vendor frameworks could eventually become practical benchmarks for agent governance, even without formal regulatory endorsement.
"Vendor frameworks can influence practical standards because developers tend to adopt the interfaces, policy templates and logging formats that are easiest to implement," Curtis said. "Over time, those choices can make their way into enterprise procurement requirements and assurance processes."
However, companies could develop competing approaches, while regulators could establish requirements that differ from industry practices. That would leave enterprises reconciling vendor safeguards with their own policies and any emerging regulation.
Because enterprise governance must extend beyond any single vendor, organizations must navigate three overlapping layers: regulators setting requirements, vendors building technical controls and enterprises deciding how much authority to give their agents.
"For enterprises moving beyond pilots, it's important to maintain an inventory of agents, their owners, connected systems and permitted actions," Curtis said. "Organizations should also distinguish between what agents can do independently, what requires approval and what should be prohibited."
Technical controls can make those decisions enforceable, but they don’t replace the enterprise's responsibility for defining them.
As AI agents take on more consequential work, the question of "safe enough" therefore extends beyond whether a technical control can contain an agent. It also depends on who defines the agent's authority, who is accountable for that decision and whether the organization can verify that it's enforcing those boundaries.
Nvidia's latest move shows that the tools for enforcing agent boundaries are taking shape. Deciding where those boundaries belong, however, remains an enterprise responsibility.