Dario Amodei is right that we need to pace the AI frontier. But pacing capability is only half the problem.
Once AI systems can use credentials, call APIs, modify infrastructure, spend money and initiate further actions, safety becomes an execution problem.
The question is no longer only: How quickly should capability advance?
It is also:
What physically prevents an AI system from executing an action it is not authorised to take?
That question leads to a different kind of AI safety architecture: runtime control.
Autonomous AI is moving from generating recommendations to executing real actions against APIs, infrastructure, credentials and money. That changes AI safety from primarily a model-behaviour problem into an execution-control problem.
In his recent essay, We Must Pace the Frontier, Anthropic CEO Dario Amodei raises a critical alarm: AI capability is advancing faster than our capacity to govern it. Citing concerns over recursive optimization and complex multi-agent behaviors, Amodei proposes pacing capability development through embedded evaluators, pre-competitive coordination among democratic labs, and eventual global agreements.
Amodei is substantially right about the problem. Capability is indeed outrunning control.
Pacing capability development addresses velocity. It does not, by itself, provide operational control. Slowing the clock buys valuable time, but pacing is a clock, not a control system. The decisive question for technology leaders and enterprise architects is straightforward: What do we build with that time?
To answer that, we must look at how the nature of AI deployment has fundamentally changed.
The Transfer of Authority
The core engineering transition facing the enterprise is not simply moving from a less capable model to a more capable model.
It is the progression along an execution chain:
A passive language model produces an output for a human to consider. An autonomous agent is authorized to act directly on infrastructure, modify databases, issue API calls, execute code, access credentials, and coordinate with other autonomous software.
When an organization moves from deploying models to deploying agents, the safety problem changes completely. The risk is no longer just that a model might generate inaccurate text or toxic language. The risk is that an autonomous execution loop makes a series of locally reasonable decisions that produce a globally unacceptable result.
It is easy to let discussions about AI control slide into science fiction tropes about sentience, malice, or power-hungry machines. That is a mistake.
An autonomous system does not need to be conscious or malicious to trigger a major incident. A stochastic planner operating over probabilistic distributions can easily cause severe operational damage simply because:
Catastrophic failure in complex software rarely stems from sudden machine intent. It stems from delegating real-world authority to a probabilistic planner without enforcing explicit runtime constraints.
Within the AI research community, immense effort goes into model alignment—using techniques like Reinforcement Learning from Human Feedback (RLHF) and Constitutional AI to train models to be helpful, honest, and harmless.
These alignment techniques are valuable and necessary. But it is vital to recognize that probabilistic behavioral alignment and deterministic operational control solve different problems.
Alignment shapes the model’s internal likelihood of proposing good decisions. Operational control governs whether the external system will permit an action to execute.
A well-aligned, highly sophisticated model can still make a mistake, hallucinate a context parameter, or misinterpret an edge-case instruction. If that model is embedded inside a loosely governed operational system, that single error can cascade into an enterprise incident. Conversely, robust authority boundaries can contain the fallout of a model error without requiring the probabilistic engine to be infallible.
We cannot wait for probabilistic models to become 100% reliable before making agentic software useful. Instead, we must build architectures where the model is permitted to propose more than it is authorized to execute.
Capability and authority must never be synonymous.
If policy promises and embedded audits evaluate whether an organization has appropriate safety intentions, runtime governance determines whether an autonomous agent can physically commit an unauthorized act at execution time.
To build meaningful runtime control into agentic systems, enterprise software must enforce five practical primitives:
Which agent, model version, and orchestration process is actually acting?
System actions must be tied to a verifiable execution identity rather than an anonymous execution worker or shared service account. For high-authority systems, that identity should be cryptographically bound where practical.
What specific act is the agent explicitly authorized to perform right now?
Authority must be explicit and non-inferable. An agent assigned to analyze a database must never infer that read-access implicitly grants schema-update or write permissions.
What hard physical and operational limits apply to execution?
Runtime boundaries must strictly govern financial spend, credential scope, network endpoints, memory consumption, execution duration, API rate limits, and downstream process creation.
What must be demonstrably true before an action is permitted, and what trace must it leave behind?
High-impact mutations—such as altering a production configuration or initiating a wire transfer—must require verifiable pre-conditions and generate tamper-evident execution logs before execution state changes.
What conditions mechanically terminate execution rather than asking the model to stop?
A control architecture must feature deterministic, out-of-band circuit breakers. If an agent exceeds a rate limit, touches an unlisted endpoint, or enters an unexpected execution loop, the system must mechanically terminate the process context, bypassing the model’s internal reasoning entirely.
A policy stating “the agent should not spend more than $500” is an intention. An architecture that mechanically rejects a $501 API transaction at the API gateway is a control system.
These principles did not emerge from theory alone.
While testing an experimental governed multi-agent system, we repeatedly found that apparently simple requirements such as “limit spend,” “use only this credential,” or “stop on disagreement” were not actually controls until they had a mechanically enforced execution path.
Several weaknesses became visible only when we tried to prove that the controls would work at runtime.
A cloud budget or anomaly alert can tell you that money has been spent, but it does not necessarily prevent the next transaction. A genuine runtime bound needs a mechanism capable of stopping execution before an authorised limit can be exceeded.
A declared scope fence is not necessarily an enforced scope fence.
We found that describing where an agent was permitted to write was different from proving that every relevant write path was mechanically constrained to that scope. The control existed in the architecture before it existed reliably in execution.
Even a valid authorisation record is incomplete unless the system defines exactly when that authority is consumed. We had to specify and enforce the point at which permission changes from available to spent, including what happens when execution fails immediately afterwards.
The recurring lesson was simple:
A policy tells an agent what it should do. A runtime control determines what it is actually able to do.
That distinction is becoming increasingly important as AI systems move from producing answers to exercising real authority.
Consider a standard enterprise scenario. A company deploys an autonomous operations agent with a high-level prompt:
“Diagnose and resolve the performance degradation on the primary customer database.”
Faced with this open-ended instruction, a capable agent might reason through a sequence of logical steps: inspect system telemetry, identify a slow query, alter an index, provision a larger database instance to relieve memory pressure, and restart the service.
From a pure problem-solving perspective, this sequence is entirely logical. But from an operational governance perspective, critical questions arise:
If these questions lack deterministic, mechanically enforced answers, the organization has not deployed a managed system. It has delegated corporate judgement and real-world authority simultaneously to a probabilistic planner.
Dario Amodei has performed a valuable public service by insisting that the AI industry confront the realities of capability scaling and control. Pacing capability growth gives society and engineers breathing room.
But pacing is a clock, not a control system.
The next great engineering challenge in artificial intelligence is not merely making models smarter, faster, or more fluent. It is building the deterministic operational control layer that sits between increasingly capable probabilistic models and production environments.
As we deploy autonomous agents deeper into enterprise workflows, the decisive factor will not be whether our models are powerful enough to plan complex tasks, but whether our systems are engineered well enough to govern their execution.
I would value hearing from engineering leaders, CTOs, and systems architects who are currently moving autonomous agents into production environments:
What explicit control mechanisms sit between your agent deciding that an action is useful and your infrastructure actually permitting that action to run?
By Mohan Iyer · 13 September 2026
Founder, Trust Before Ship. Building and testing governed multi-agent execution systems.