The whole design, in writing
Every step of the build above, written out: the problem each piece solves, the option that was taken and the ones that were not, the numbers, and how it fails in production.
The big idea
A control plane, not a coordination pattern
This isn't about how one agent breaks a task into subtasks, or how several agents cooperate on one job — that's multi-agent orchestration, a different design entirely. This is about the boring, essential platform question an organization hits the moment it has more than a handful of agents running in production: who owns each one, what can it actually do, what's it allowed to spend, and how do you stop it RIGHT NOW if it starts misbehaving?
We'll build the fleet-management layer: a registry so nothing runs unaccounted-for, server-enforced tool permissions so an agent can't just do whatever its prompt claims it should, hard budgets checked continuously, an out-of-band kill switch, and full execution tracing.
What the new pieces do
- Operatorclient
- The team deploying and operating agents in production — wants visibility and control across potentially hundreds of independent agent deployments.
Step 1 · The baseline
Agents as ad-hoc scripts
The organic starting point: someone writes an agent script, deploys it wherever, and it runs. Another team does the same. A few months later, dozens of these exist. What's the actual operational problem?
Dozens of independently-deployed agent scripts exist across an org, with no central record. What's the core problem?
"Each one works" is exactly the assumption that breaks down the first time one of them doesn't — with no central record, nobody even knows the full list of what's running, let alone who to page or how to stop it.
This is a fleet-visibility and governance problem: without a central registry, an organization genuinely cannot answer basic operational questions — "what agents exist," "what can this one do," "who's responsible for it," "how do I stop it" — which becomes actively dangerous once agents can call real tools and spend real money.
Performance of any individual agent is a separate, secondary concern — the immediate problem is the complete lack of fleet-wide visibility and control, regardless of how fast any one agent happens to run.
With no central registry, nobody has a complete picture of what agents exist, what they can do, who owns them, or how to control them — a governance blind spot that compounds as the fleet grows and as agents gain access to real tools and real budgets.
Why this piece earns its place
Before a registry exists you still have an inventory — it is just the wrong one. The list you can actually produce today is the provider bill and the set of API keys, and neither is a list of agents. One key gets pasted into three scripts, so spend cannot be attributed to any of them, and revoking it to stop one thing breaks two others nobody knew were there. The de facto identity of an agent is a credential somebody copied, which is why the first real incident tends to end in an org-wide key rotation instead of a targeted stop. Worth being honest about when this does not pay yet: one team, one repo, a handful of agents everyone can name in a meeting. What usually ends that state is not a headcount of agents, it is the first resignation. An agent outlives the engineer who wrote it, what it was for leaves with them, and nobody who stays is willing to be the person who turned it off — so unowned agents accumulate, because switching one off is a risk somebody wears and leaving it running is not. That is also why the owner field is a team and not a name. And plan the adoption path now. An unregistered run should keep working while being reported for a while, because a platform whose first act is breaking everyone's deploy does not get adopted.
What the new pieces do
- Agent Runtimeservice
- The execution environment for a single agent invocation — starts as an unmanaged, ad-hoc script; becomes governed by every control the platform adds.
Step 2 · A central registry
Every agent, versioned, owned
Fix the blind spot: every agent has to be registered before it can run.
Introduce an Agent Registry: every agent gets an id, an owning team, a declared version, and a declared set of tools it needs. Deploying a new version doesn't silently overwrite the old one in place — versions coexist in the registry, which is what makes staged rollout (Step 7) possible later. The Control Plane API becomes the single front door for registration and deployment, replacing ad-hoc scripts running wherever someone happened to put them.
Why this piece earns its place
Decide early what a version actually contains, because for an agent it is not just code. The prompt, the model id and its parameters, the declared tool list and the code each change behaviour on their own, and if any one of them can move without the version moving, every later control loses its meaning: the canary compares two things that are no longer distinct, and a trace from last Tuesday cannot be reproduced. Make a registry entry immutable, and make a change mint a new version rather than editing one in place. The second decision is organizational, not technical — who may edit an agent's declared tool list. If the team that writes the prompt can also grant itself a tool, the allowlist is self-service and the gate built in the next step is decorative. Route that one field through review while leaving prompt changes fast. The registry is also read on the hot path now, so it will end up cached — put that copy in the enforcement service, never in the agent's own process, and price the staleness out loud: a cached allowlist means a revoked tool stops working after the window, not at the next call. Make revocation a write that invalidates rather than a TTL you wait out. An owner field can sit stale and nobody is harmed; the tool list is the one where the window is the gap between revoked and actually revoked.
What the new pieces do
- Control Plane APIgateway
- The single front door for managing every agent in the fleet — deployment, permissions, budgets, and emergency controls all flow through here.
- Agent Registrydata
- The catalog of every agent: who owns it, which version is live, what tools it's declared to need — the platform's source of truth for "what's actually running."
Step 3 · Server-enforced tool permissions
Don't trust the prompt to self-restrict
An agent's system prompt says "you may only use the search tool and the calculator." Is that instruction, by itself, an actual security boundary?
An agent is instructed via its prompt to only use certain tools. Is that a real security boundary?
A prompt is a strong INFLUENCE on typical behavior, but it's not an enforced boundary — a sufficiently unusual input, a prompt injection, or the model simply making a mistake can lead it to attempt a tool call the prompt told it not to make, with nothing actually stopping the call from executing.
Server-side enforcement means even if the model attempts an out-of-scope tool call — through injection, a bug, or unexpected behavior — the Tool Permission Gate rejects the call before it executes, regardless of what the agent's own reasoning or prompt intended. The security boundary lives in the platform, not in the model's good behavior.
"Technically available" is exactly the problem — if every agent has blanket access to every internal tool and API by default, there's no boundary at all beyond the model choosing to behave, which is not a security guarantee.
Add a Tool Permission Gate: every tool call an agent attempts is checked, server-side, against that specific agent's declared allowlist (from its registry entry) — enforced at call time, not just requested via prompt. An agent attempting a tool outside its scope gets rejected by the platform, regardless of why it tried.
Why this piece earns its place
A tool-name allowlist is the first cut, not the whole boundary. send_email is one entry, and the difference between mailing a colleague and mailing the customer list lives entirely in the arguments — so the gate has to read them, which means every tool needs a declared shape and a permission is a rule over that shape, not a boolean. The second thing to settle is whose authority a call carries. If the agent executes as whoever triggered it, its real reach is that person's access and it changes per caller — the same agent is harmless for one user and able to read payroll for another, and the allowlist describes neither of them. Give the agent an identity of its own, scoped to its allowlist, so the gate's answer and the call's reach describe the same thing. Where the credential behind that identity physically lives is a separate question, and Step 8 argues it should never be inside the agent. Then be deliberate about how a rejection comes back. Return it as an ordinary tool result the model can read — denied, and why — and it will try another route. Raise it as a crash and a stubborn agent retries the same forbidden call until the budget check halts the run, which files a permissions problem under the wrong alert.
What the new pieces do
- Tool Permission Gateservice
- Checks every tool call an agent attempts against its declared allowlist — enforced by the platform, not just requested politely in a prompt.
Step 4 · Hard cost budgets
A loop can spiral cost fast — bound it
An agent that calls an LLM in a loop (retrying, re-planning, chaining tool calls) can, if something goes wrong, spiral into dozens or hundreds of extra calls before anyone notices — real, immediate cost. Should budget enforcement happen only after a run completes?
A runaway agent loop could burn far more cost than intended before anyone notices. When should budget be checked?
By the time a post-hoc report shows the total cost, the money is already spent — a report catches the problem AFTER the damage, which doesn't bound the actual blast radius of a runaway loop at all.
Checking budget as part of every step (not just at the end) means a runaway loop is caught and stopped mid-flight, the moment it crosses its limit — bounding actual spend to close to the configured cap, rather than however much accumulated before someone happened to look at a report.
A budget that isn't actually enforced with a hard stop is just a suggestion — it does nothing to bound a genuine runaway loop, which is exactly the failure mode this control needs to prevent.
Enforce budgets continuously, per step — the Budget Enforcer tracks cumulative cost, tokens, and tool-calls for a run in real time and halts execution immediately the moment it crosses its configured cap, rather than only reporting cost after the fact.
Why this piece earns its place
The check is easy; the counter is the design. It has to live outside the agent process and survive a restart, because the obvious place to keep a running total is in memory beside the loop — and then a crash, a retry, or a second replica starts again at zero, which makes the expensive path the free one. Cost also arrives late: you only know what a call cost once it returns, so a check that reads settled spend is permanently one call behind. Reserve an estimate before the call and reconcile against real usage after, and the overshoot is bounded to roughly one step. The operational half matters more than either. A cap hit has to be its own outcome — alerted and traced as such, not folded into generic run failures. Otherwise an agent that quietly clips its ceiling on a steady share of its runs reads as a flaky agent, and the reflex fix is raising the cap rather than finding the loop. Keep the cap in the registry per agent, and refuse the obvious escape hatch of a run asking for more in its own request: a run that can raise its own ceiling does not have one. Work that genuinely needs a larger cap is a second registered configuration, granted the way a tool is granted, so the raise is recorded before the run starts and by someone other than the caller.
- per-step checknot end-of-run only
- hard halton cap breach
- bounded blast radiusregardless of loop length
What the new pieces do
- Budget Enforcerservice
- Tracks cumulative cost/tokens/tool-calls for a run in real time and halts it the moment it crosses its configured cap.
Step 5 · The kill switch
Stop it now, without a code deploy
An agent in production starts behaving badly — not necessarily exceeding its budget, just doing something wrong (bad tool-call patterns, unexpected outputs). An operator needs to stop it immediately. Should that require a code change and redeploy?
Provide a Kill Switch at the Control Plane level: an operator can halt a specific agent (or the entire fleet) instantly, with no code deploy and no dependency on the agent's own code cooperating. This is deliberately an out-of-band control — if an agent is misbehaving badly enough to need killing, you cannot assume its own code will gracefully respond to a "please stop" instruction.
Why this piece earns its place
Two details decide whether this holds on the day it is needed. The first is what the runtime does when it cannot reach the kill state at all. Treating unreachable as not killed is a switch that fails open under exactly the conditions — a bad deploy, an overloaded platform — that tend to accompany the incident. Cache the last known state, fail closed past a short grace window, and accept that a platform outage stops the fleet. The second is granularity. Kill by agent id is the obvious axis, but the ones you want at 3am are kill this version, kill anything touching this tool, and kill everything for this tenant, and each has to be expressible before the incident rather than invented during it. Then the parts nobody writes down. A kill has no automatic release, so decide what happens to the work piling up behind it: drained, dropped, or parked for replay. One on-call engineer must be able to pull it alone, since any approval chain turns it back into a deploy. And rehearse the fleet-wide one — an untested big red button is a belief, not a control.
What the new pieces do
- Kill Switchcontrol
- An operator-triggered, platform-level halt for a specific agent or the whole fleet — works regardless of what the agent's own code is doing.
Step 6 · Full execution tracing
Debugging needs the whole run, not the final answer
An agent produced a wrong or strange result. Understanding why requires seeing every step it took to get there.
Record a full Execution Trace for every agent run: every tool call and its arguments, every intermediate output, every cost increment, in order. When something goes wrong, an operator can replay exactly what the agent did and why, rather than only seeing a final answer with no visibility into the reasoning or tool-call chain that produced it.
Why this piece earns its place
A trace is not a log line with more fields. It is a verbatim copy of everything the agent read and wrote, which means customer data and, if you are careless, tokens and keys land in an observability store with different access rules and a different retention clock than the systems they came from. Redact at write time, not at read time — a trace you have to remember to filter before opening is one you will not open during an incident. Volume forces the next decision. Traces are far larger per run than a request log, so you will sample; sample the successes only and keep every run that ended in a cap hit, a permission denial or a kill, because those are the ones anyone actually reads. Then stamp each trace with the things that cannot be reconstructed afterwards: the agent id, the exact registry version that ran, and the tools it was allowed at that moment. Without the version on the trace, a period of rollout is one undifferentiated pile of runs that nobody can split by version later, and the allowlist has probably moved since, so a denial you are reading back is explained by a rule you can no longer see. And one trace id has to span nested tool calls and any sub-agent, or the trail breaks at exactly the boundary that was interesting.
What the new pieces do
- Execution Tracesdata
- A full record of every agent run — every tool call, every input/output, every dollar spent — because debugging "why did the agent do that" needs the whole trace, not just a final answer log.
Step 7 · Canary rollout
A new agent version earns full traffic, it doesn't start with it
An agent's behavior changed with a new prompt, a new model, or new tool logic. Should the new version immediately handle 100% of that agent's traffic?
A new version of an agent is ready to deploy. Should it immediately take over all traffic for that agent?
Pre-deploy testing can't fully replicate the diversity of real production inputs, and agent behavior specifically (unlike a simple deterministic service) can shift in subtle, hard-to-test-for ways with a new prompt or model — a full, immediate cutover risks a much larger blast radius if something's wrong than a staged rollout would.
A canary rollout, using the same versioning the Registry already tracks, limits exposure to a new agent version: a small traffic slice reveals real-world behavior differences (cost spikes, unexpected tool-call patterns, quality regressions) while the blast radius of a problem stays small, before the new version earns full traffic.
Manually choosing between agent versions isn't a practical or scalable approach for most agent products — users generally aren't equipped to evaluate agent-version quality themselves, and this doesn't actually solve the "how do we safely validate a new version" problem.
Use the Registry's versioning to run a canary rollout: route a small percentage of an agent's traffic to the new version, compare error rate, cost, and quality signals against the previous (still-live) version, and only promote to 100% once it's proven out — with an immediate rollback path (routing traffic back to the prior version) if it isn't.
Why this piece earns its place
Routing a percentage is the easy half. The hard half is that the metric which normally decides a canary barely moves here — both versions return a confident answer, and the regression is in what the answer was and what it cost. So the decision rule has to exist in numbers before the rollout starts, and its inputs have to be measurable at canary volume. Watch cost per run rather than total spend: on a slice that small, a per-run cost regression disappears into the noise of the total bill. And slice by conversation or user rather than per request, because a session that flips versions mid-thread produces a transcript that is neither version's behaviour and that nobody can debug afterwards. The promotion side needs the same discipline as the rollback. Rollback is a routing change and therefore instant, but only while the previous version's artifact is still deployed and untouched — so do not reclaim it the moment the new one reaches full traffic. Give it a defined bake period and name who decides, or the canary quietly becomes a permanent state — a version parked on a small slice that nobody promotes and nobody retires, with the fleet carrying two live behaviours per agent and no one accountable for which is which.
Step 8 · The sharp edges
Enforcement has to be real, not advisory
Every control built so far only matters if it's actually, robustly enforced under real adversarial or buggy conditions — a permission check that can be bypassed, or a budget check that runs too infrequently, isn't a real control.
Tool permissions are checked server-side on every single call, never trusted to prompt instructions alone. Budget enforcement runs per step, not just at run-end, bounding a runaway loop's actual cost. The kill switch is out-of-band, independent of the misbehaving agent's own code. Together, these mean the platform's controls hold even when an individual agent is buggy, compromised, or adversarially prompted — which is exactly the condition under which they matter most.
Why this piece earns its place
Every control on this page rests on one assumption worth saying out loud: the agent cannot route around it. If the runtime is a library the agent imports, then a pinned old version, a direct call to the vendor SDK, or a second process started by the same team bypasses all of it while still looking governed in the registry. The structural fix is to put enforcement where the credential is — the agent never holds the token for the identity it runs as, it asks the platform to make the call and the platform holds the secret. A check you can skip is advice; a check you cannot get a credential past is a boundary. That choice has a price, and an interviewer will want to hear it. Every control now needs a declared answer for the moment the platform has none, and what decides it is what that control protects: a check standing between an agent and an irreversible action fails closed, so a permission lookup with no answer is a refused call, while a component that only records fails open — a tracing backend that is down must never be the reason a run stops. Then name the break-glass path before you need it: who may run work without the platform, for how long, and how it is reviewed afterwards. The version nobody wrote down is an engineer with a copied key and a deadline.
You did it
You just designed an agent control plane.
Everything you assembled, in order
- A central registry (id, owner, version, declared tools) is the prerequisite for every other control — you can't govern agents the platform doesn't know exist.
- Tool permissions are enforced server-side, at call time, against a declared allowlist — never trusted to prompt instructions alone.
- Cost/resource budgets are checked continuously, per step, so a runaway loop is halted mid-flight rather than only reported on after the fact.
- A kill switch is an out-of-band, platform-level halt, independent of the misbehaving agent's own code cooperating.
- Full execution tracing (every tool call, input, output, cost increment) is what actually makes debugging agent behavior possible.
- New agent versions earn full traffic through a canary rollout, monitored on error rate, cost, and quality — not an immediate full cutover.
