The big idea

A control plane, not a coordination pattern

This isn't about how one agent breaks a task into subtasks, or how several agents cooperate on one job — that's multi-agent orchestration, a different design entirely. This is about the boring, essential platform question an organization hits the moment it has more than a handful of agents running in production: who owns each one, what can it actually do, what's it allowed to spend, and how do you stop it RIGHT NOW if it starts misbehaving?

We'll build the fleet-management layer: a registry so nothing runs unaccounted-for, server-enforced tool permissions so an agent can't just do whatever its prompt claims it should, hard budgets checked continuously, an out-of-band kill switch, and full execution tracing.

How to read this: Each step opens with a real design decision — make the call before I show you what ships. Watch the diagram grow, hover the boxes, and at the end let a loop spiral toward its cost cap and trip the kill switch to see the platform-level controls actually engage. Hit Begin.

Step 1 · The baseline

Agents as ad-hoc scripts

The organic starting point: someone writes an agent script, deploys it wherever, and it runs. Another team does the same. A few months later, dozens of these exist. What's the actual operational problem?

Design decision: Dozens of independently-deployed agent scripts exist across an org, with no central record. What's the core problem?

The call: Nobody has a complete picture of what agents are actually running in production, who owns each one, what tools/APIs they can call, or how to control them centrally — an operational and security blind spot that only gets worse as the fleet grows. — This is a fleet-visibility and governance problem: without a central registry, an organization genuinely cannot answer basic operational questions — "what agents exist," "what can this one do," "who's responsible for it," "how do I stop it" — which becomes actively dangerous once agents can call real tools and spend real money.

With no central registry, nobody has a complete picture of what agents exist, what they can do, who owns them, or how to control them — a governance blind spot that compounds as the fleet grows and as agents gain access to real tools and real budgets.

You can't govern what you can't see: This is the foundational reason a control plane exists at all: every subsequent capability (permissions, budgets, kill switches, tracing) depends on the platform first knowing, definitively, what agents exist and where they're running.

Step 2 · A central registry

Every agent, versioned, owned

Fix the blind spot: every agent has to be registered before it can run.

Introduce an Agent Registry: every agent gets an id, an owning team, a declared version, and a declared set of tools it needs. Deploying a new version doesn't silently overwrite the old one in place — versions coexist in the registry, which is what makes staged rollout (Step 7) possible later. The Control Plane API becomes the single front door for registration and deployment, replacing ad-hoc scripts running wherever someone happened to put them.

A registry is the prerequisite for every other control: You can't enforce a tool permission, a budget, or a kill switch on an agent the platform doesn't know exists. The registry is the foundation everything else in this design is built on top of.

Step 3 · Server-enforced tool permissions

Don't trust the prompt to self-restrict

An agent's system prompt says "you may only use the search tool and the calculator." Is that instruction, by itself, an actual security boundary?

Design decision: An agent is instructed via its prompt to only use certain tools. Is that a real security boundary?

The call: No — the platform must enforce a server-side allowlist of which specific tools/APIs each agent is permitted to call, checked at the moment of every tool invocation, independent of what the prompt says. — Server-side enforcement means even if the model attempts an out-of-scope tool call — through injection, a bug, or unexpected behavior — the Tool Permission Gate rejects the call before it executes, regardless of what the agent's own reasoning or prompt intended. The security boundary lives in the platform, not in the model's good behavior.

Add a Tool Permission Gate: every tool call an agent attempts is checked, server-side, against that specific agent's declared allowlist (from its registry entry) — enforced at call time, not just requested via prompt. An agent attempting a tool outside its scope gets rejected by the platform, regardless of why it tried.

Enforce boundaries in the platform, not in the model's good behavior: A prompt instruction is a strong nudge, not a guarantee — LLMs can be pushed off-script by injection, edge cases, or plain mistakes. Any boundary that actually needs to hold has to be enforced by code that isn't itself the thing being instructed.

Step 4 · Hard cost budgets

A loop can spiral cost fast — bound it

An agent that calls an LLM in a loop (retrying, re-planning, chaining tool calls) can, if something goes wrong, spiral into dozens or hundreds of extra calls before anyone notices — real, immediate cost. Should budget enforcement happen only after a run completes?

Design decision: A runaway agent loop could burn far more cost than intended before anyone notices. When should budget be checked?

The call: Continuously, per step, during the run itself — the Budget Enforcer tracks cumulative cost/tokens/tool-calls in real time and halts the run the instant it crosses its configured cap. — Checking budget as part of every step (not just at the end) means a runaway loop is caught and stopped mid-flight, the moment it crosses its limit — bounding actual spend to close to the configured cap, rather than however much accumulated before someone happened to look at a report.

Enforce budgets continuously, per step — the Budget Enforcer tracks cumulative cost, tokens, and tool-calls for a run in real time and halts execution immediately the moment it crosses its configured cap, rather than only reporting cost after the fact.

A limit that's only checked at the end isn't a limit: For any resource that can be consumed rapidly by a runaway process (cost, here — but the same logic applies to memory, CPU, or API rate elsewhere), the enforcement mechanism has to run continuously alongside the consumption, not as a retrospective audit.

Step 5 · The kill switch

Stop it now, without a code deploy

An agent in production starts behaving badly — not necessarily exceeding its budget, just doing something wrong (bad tool-call patterns, unexpected outputs). An operator needs to stop it immediately. Should that require a code change and redeploy?

Provide a Kill Switch at the Control Plane level: an operator can halt a specific agent (or the entire fleet) instantly, with no code deploy and no dependency on the agent's own code cooperating. This is deliberately an out-of-band control — if an agent is misbehaving badly enough to need killing, you cannot assume its own code will gracefully respond to a "please stop" instruction.

Emergency controls must not depend on the thing they control: A kill switch that relies on the misbehaving agent's own code to check a flag and stop itself is a kill switch that can fail exactly when it's needed most. The control has to live entirely outside the agent, enforced by the platform (e.g. the runtime refuses to schedule/continue any invocation of a killed agent id).

Step 6 · Full execution tracing

Debugging needs the whole run, not the final answer

An agent produced a wrong or strange result. Understanding why requires seeing every step it took to get there.

Record a full Execution Trace for every agent run: every tool call and its arguments, every intermediate output, every cost increment, in order. When something goes wrong, an operator can replay exactly what the agent did and why, rather than only seeing a final answer with no visibility into the reasoning or tool-call chain that produced it.

Observability for agents means the whole decision trail: Unlike a traditional service where a request/response log pair is often enough, an agent's BEHAVIOR (which tools it chose, in what order, based on what intermediate results) is itself the thing that most often needs debugging — the trace has to capture that trail, not just the endpoint.

Step 7 · Canary rollout

A new agent version earns full traffic, it doesn't start with it

An agent's behavior changed with a new prompt, a new model, or new tool logic. Should the new version immediately handle 100% of that agent's traffic?

Design decision: A new version of an agent is ready to deploy. Should it immediately take over all traffic for that agent?

The call: Route a small percentage of traffic to the new version first, monitor its error rate, cost, and quality metrics against the previous version, and only promote it to 100% once it's proven itself. — A canary rollout, using the same versioning the Registry already tracks, limits exposure to a new agent version: a small traffic slice reveals real-world behavior differences (cost spikes, unexpected tool-call patterns, quality regressions) while the blast radius of a problem stays small, before the new version earns full traffic.

Use the Registry's versioning to run a canary rollout: route a small percentage of an agent's traffic to the new version, compare error rate, cost, and quality signals against the previous (still-live) version, and only promote to 100% once it's proven out — with an immediate rollback path (routing traffic back to the prior version) if it isn't.

Staged rollout, applied to agent BEHAVIOR quality: This is the standard canary-deployment pattern from traditional service rollouts, applied to a domain where "did this break" isn't just uptime/error-rate — it also includes cost behavior and output quality, both of which can regress in a new agent version even when the service itself stays technically up.

Step 8 · The sharp edges

Enforcement has to be real, not advisory

Every control built so far only matters if it's actually, robustly enforced under real adversarial or buggy conditions — a permission check that can be bypassed, or a budget check that runs too infrequently, isn't a real control.

Tool permissions are checked server-side on every single call, never trusted to prompt instructions alone. Budget enforcement runs per step, not just at run-end, bounding a runaway loop's actual cost. The kill switch is out-of-band, independent of the misbehaving agent's own code. Together, these mean the platform's controls hold even when an individual agent is buggy, compromised, or adversarially prompted — which is exactly the condition under which they matter most.

Design for the unhappy path: A registry that's advisory, permissions that are prompt-only, a budget checked too late, or a kill switch the agent could theoretically ignore — every one of these would only work when nothing's actually wrong. A control plane that only works for well-behaved agents isn't a control plane; the whole point is holding when something ISN'T well-behaved.

You did it

You just designed an agent control plane.

• A —
• T — o
• C — o
• A —
• F — u
• N — e