The whole design, in writing
Every step of the build above, written out: the problem each piece solves, the option that was taken and the ones that were not, the numbers, and how it fails in production.
The big idea
What is an agent harness?
An engineer with a coding agent is faster at the task in front of them. An engineering organisation with agents and no harness is faster at producing work nobody has verified — and a growing pile of plausible diffs is not throughput.
Build the system around the agent: decompose requests into units whose success a machine can confirm, constrain capability to a designed tool surface, sandbox execution so failure is cheap, verify outside the agent’s reach, and route review by risk. The job moves from writing feature code to writing the system that writes it.
Step 1 · The skeleton
An agent, a repo, a task
The naive version: an engineer describes a change, an agent edits files, the engineer reads the diff. It works, and it does not scale — because the engineer is now the entire verification layer, and reading a plausible diff carefully is slower than writing it.
Stand up a Planner that hands tasks to an Agent and a human who supervises rather than authors. Everything that follows makes the agent’s output bounded, checkable, and cheap to be wrong.
Why this piece earns its place
A run needs an owner the moment it can fail in more than one way, and the planner is the only component positioned to be it. An agent that cannot succeed does not stop — it keeps calling tools, re-reading the same file, and spending tokens on a task that was impossible the day it was written. So every run carries a step and token budget, and that budget belongs to the task, not to the config: a typo fix and a wide refactor cannot share a ceiling without one of them being wrong. Budget exhaustion also needs to be its own terminal state, distinct from failed verification, because the follow-ups differ — one says the unit was decomposed badly, the other says the change was wrong. This is also the only component that knows how many runs are in flight, so the ceiling on concurrent agents sits here, along with the decision to re-dispatch a spent unit rather than let the agent retry inside its own loop, where a second attempt is indistinguishable from the first. Skip all of it while you are one engineer with one agent; you are the lifecycle. Once there are queues, retries and abandoned runs, that policy exists whether or not anyone designed it, and undesigned it lives in slightly different copies inside every caller.
What the new pieces do
- Engineerclient
- No longer writing every line — specifying, reviewing the diffs that matter, and improving the harness when a class of failure repeats. The job moves from producing code to producing the system that produces it.
- Agentservice
- The model in a loop: read context, call a tool, observe, repeat. Capable exactly to the extent the tool surface allows, and no further.
Step 2 · Bound it
Tool surface and sandbox
Give an agent a shell and you have given it every capability at once: your credentials, your network, your production. The blast radius is unbounded, the audit trail is unreadable, and you cannot answer a security reviewer’s first question about what it can do.
What is the strongest constraint on what an agent can do?
A prompt is guidance, not a control. It fails under an unusual input, an injected instruction in a file it reads, or a long enough loop — and none of those are edge cases in practice.
A real control and the wrong layer. By then the agent has already run whatever it ran; the diff shows you the code it wrote, not the requests it made or the data it touched.
Capability comes only from tools. If there is no tool to delete a bucket, the agent cannot delete a bucket — regardless of what it decides, or what a document it reads tells it to do.
Give the agent a designed Tool Surface — read, write, test, search, open a pull request — with no general shell, and run every task in its own Sandbox: an ephemeral worktree and container with scoped credentials, discarded on failure.
Why this piece earns its place
A tool’s output is model input, so tool design is context design. A search tool that returns every match in a large repository does not fail — it quietly evicts the agent’s earlier reasoning from the window, and the run degrades into re-deriving what it already knew. So every tool returns a bounded page, and truncation has to be visible: the tool states how much it withheld, because an agent handed a silent subset concludes the thing does not exist and writes a second copy of it. Error text is the same story in reverse, since it is the agent’s only feedback channel — a tool that fails with permission denied produces a retry loop, while one that names the missing capability produces a correction. Budget the sandbox too: a cold container plus a dependency install can cost more wall-clock than the agent’s own work, so the start-up is worth engineering — pre-built images, dependencies baked in, containers made ready before a task arrives. Warm has to mean ready, never reused: a pooled workspace is still handed to exactly one task and destroyed with it, and sharing one across tasks buys a few seconds and rebuilds the corruption this step exists to prevent. A broader surface is defensible for read-only agents, where nothing they do is durable.
- no shelldesigned tools only
- one workspaceper task
- scoped credsnever production
What the new pieces do
- Tool Surfaceservice
- Read file, write file, run tests, search the repo, open a pull request. Explicitly designed, individually logged, and deliberately without a general shell — capability is granted, never assumed.
- Sandboxservice
- One ephemeral workspace per task — its own worktree, its own container, its own scoped credentials — discarded on failure. Makes a bad run cost nothing and lets many agents run at once.
Back of the envelope
- designed tool surface
- capability is granted, not assumed
- ephemeral workspace per task
- discardable failure
- scoped, short-lived credentials
- no production access
- every tool call logged
- with arguments and result size
Step 3 · The real work
Decompose until success is checkable
“Add multi-currency support” is a request an agent will happily attempt and nobody can verify without reading everything it wrote. The failure is not the model — it is handing it a task with no machine-checkable definition of done.
The Planner splits a Request into units small enough that each carries an explicit acceptance condition: a failing test that must pass, a schema that must validate, an eval that must clear. If it cannot produce one, the task goes back to a human.
Why this piece earns its place
Two forces set the size of a unit and only one of them is obvious. The ceiling is verifiability. The floor is fixed cost: every unit pays for a fresh workspace, a context retrieval and a full verification run, so decomposing past that point spends more on overhead than on the work, and starves each unit of the surrounding code that made the change coherent in the first place. Units are also not independent, and the dependency graph is what teams discover late. If one unit needs an interface another has not built yet, you either sequence them and lose the parallelism that justified the harness, or you fix the contract up front and let both run against it — decide that while you are decomposing, not while you are merging. Then decide the harder one: what happens when five units pass and two do not. If units merge independently, a half-built feature is now on the main branch behind nothing, so the merge target and the flag are chosen at the same moment the units are written.
- acceptance conditionper unit
- no condition→ back to a human
- tests firstthen the agent
What the new pieces do
- Requestclient
- A bug report, a customer request, a backlog item. Almost never specified well enough for an agent — turning it into verifiable units is the harness’s real work.
- Plannerbackend
- Splits a request into tasks small enough that success is machine-checkable, each with an explicit acceptance condition. If it cannot produce one, the task goes back to a human rather than forward to an agent.
Back of the envelope
- each unit has a machine-checkable done
- test, schema or eval
- unverifiable tasks are refused
- not attempted
- the spec is the deliverable
- the engineer’s highest-value output
Step 4 · The boundary
Verification the agent cannot reach
The agent reports that it fixed the bug and the tests pass. Both statements came from the same system that wrote the code — and if the agent can edit the test, the test measures nothing at all.
The agent says its change passes all tests. What do you do with that claim?
The claim and the code come from the same source. A model asked whether it succeeded reports success at a rate uncorrelated with actual success, and it is the subtle failures it misses.
The agent’s report is a hint; the external run is the evidence. This boundary is the single most important line in the whole design, and it is the one most often relaxed for speed.
Useful as an extra signal and not a substitute — two models sharing a failure mode agree with each other confidently. A deterministic check is not a matter of opinion.
Run Verification outside the agent’s reach: tests, type checks, lint, build and evals on infrastructure it cannot configure or bypass. The agent’s own claim is never the evidence.
Why this piece earns its place
Where verification runs is settled. What it measures is not, and that is the half that bites. An agent optimises against whatever the gate checks, so a thin suite does not merely fail to catch bad work — it steers the agent toward work that satisfies a thin suite, again and again in the same module, which from the outside reads as the model being weak in that area. The gate is a specification whether or not anyone wrote it as one. Flakiness also changes character. A human re-runs a flaky test and shrugs; an agent takes the red as ground truth and starts repairing code that was already correct, or retries until it goes green and merges something that never worked. Quarantining flakes stops being hygiene and becomes a prerequisite. And verification latency sets the agent’s iteration rate, so a slow suite does not just delay a merge — it caps how many corrections the agent gets to make before its budget is spent. The fix belongs on the runner rather than in the loop: shard the suite, order it so the checks nearest the diff report first, and stream that first red back early. The feedback arrives sooner, and the thing selecting and executing the checks is still something the agent cannot touch.
- external runneragent cannot configure
- deterministicnot a second opinion
- claim ≠ evidencealways re-run
What the new pieces do
- Verificationservice
- Tests, type checks, lint, build and evals, executed on a runner the agent cannot configure, skip or edit. The agent’s own report of success is never the evidence.
Back of the envelope
- verification outside the agent’s reach
- the one hard boundary
- tests, types, lint, build, evals
- all of them, every run
- self-report treated as a hint
- never as proof
Step 5 · Supervision economics
A review queue routed by risk
Review every diff and the harness is slower than doing the work by hand. Review none and an unsupervised agent is editing your schema. Both failure modes are common, and they are the same mistake: treating all changes as equally risky.
Route the Review Queue by blast radius. Auto-merge reversible, well-covered changes in low-risk areas. Require a named human for schema migrations, authentication, money, deletions, and anything touching a customer environment.
Why this piece earns its place
Routing by risk makes the queue survivable; the next failure is the reviewer. Reviewing a colleague’s pull request works because you shared the design conversation that preceded it. With an agent diff there is no prior, so a change that arrives as code alone forces the reviewer to reconstruct intent from the implementation — the slow path this whole system exists to avoid. Send the evidence with it: the acceptance condition the unit was given, the verification output, the context it was grounded in. The second-order failure is the one no dashboard shows. Once the queue grows past what a person can genuinely read, the approval rate drifts toward everything while throughput, latency and merge rate all keep looking excellent. So watch the rejection rate, per reviewer: someone who never rejects is either being handed only safe work or has stopped reading, and those two need different repairs. In a pre-production repository with no customers you can skip most of this and lean on revert — the day customers arrive, that stops being true quietly.
- auto-mergereversible + covered
- named humanschema · auth · money
- queue by risknot by volume
What the new pieces do
- Review Queuecache
- Routes by blast radius rather than by volume: reversible, well-covered changes auto-merge; schema, auth, money and anything touching a customer environment require a named human.
Back of the envelope
- blast-radius classification per path
- decides the route
- auto-merge needs coverage evidence
- not just green tests
- customer-environment changes always reviewed
- no exceptions
Step 6 · Ground it
Context: code that belongs in this repo
Without repository context an agent writes plausible code in the wrong style, re-implements a helper that already exists, ignores a convention three teams depend on, and reintroduces a bug someone fixed deliberately last year.
Retrieve Repo Context per task: conventions, ownership, prior decisions, the relevant slice of the codebase, and the reasons behind them. Ground the agent in what already exists before it writes anything new.
Why this piece earns its place
Written conventions are a second source of truth beside the code, and the two drift. When they do the agent follows the old rule confidently and produces something consistent enough to pass review, until someone who remembers says “we stopped doing that last year”. So treat prose as the fallback and move every convention that can be made executable into a lint rule, a type or a codemod — which moves it out of context, where it is a suggestion, and into verification, where it is enforced and cannot rot without failing loudly. The other constraint is budget. Retrieved context competes with the agent’s own working state for one window, so a generous retrieval leaves less room to reason and pushes the earliest instructions away from where the model attends. That makes context the first thing anyone trims when runs get expensive, usually without measuring what it was buying. Which is why the thin case has to be designed rather than discovered: an agent handed nothing relevant still writes code, and that run looks exactly like a good one until somebody reads it. A retrieval that comes back weak should push the task into the reviewed class or hand it back, on the principle that refuses an unverifiable unit: thin retrieval says this repository has not been described well enough for the change being asked of it.
- conventionsretrieved per task
- prior decisionsand their reasons
- retrieved textis data, not instruction
What the new pieces do
- Repo Contextdata
- Conventions, ownership, prior decisions and the shape of the codebase — retrieved per task. The difference between an agent that writes plausible code and one that writes code that belongs in this repository.
Back of the envelope
- task-scoped retrieval
- not the whole repository
- decisions and their rationale
- so the agent does not undo them
- retrieved content delimited
- injection is a live risk in a repo
Step 7 · Answerable
The ledger
Six months on, a customer asks why a behaviour changed, or an auditor asks who approved a change to a calculation. “An agent did it” is not an answer anyone accepts, and reconstructing it from git history alone is impossible.
Record every run in a Ledger: prompt, model version, retrieved context, every tool call, the diff, verification results, and the human who approved it. Retain it like any other production audit trail.
Why this piece earns its place
The audit case gets the ledger funded; the second use is what makes it worth the storage. Every run with its verification result is a labelled example — this task shape, this context, these tool calls, passed or failed — so the ledger turns the agent is bad at migrations from an anecdote into a query, and it is what tells you which tool to build next rather than guessing. Three practical notes. Write it as the run happens rather than at the end, because the records worth reading are the runs that looped, timed out or were cancelled, and none of those reaches a tidy final step. Keep the bodies out of the record: a read tool returns whole files and a long run returns many of them, so hold hashes and references and put the payloads in cheap object storage, or the ledger costs more to keep than the work it describes. And it inherits the access controls of the code it quotes — prompts, retrieved source and diffs together are a way to read a repository nobody granted you, so this is not a log everyone can grep. Underneath all of it: the harness writes the record, never the agent. A history the agent supplies has exactly the standing of its own report of success.
- model + prompt versionper run
- every tool callrecorded
- approvernamed
What the new pieces do
- Ledgerdata
- Prompt, model version, every tool call, the diff, verification results and the approving human. Retained like any other production audit trail, because someone will eventually ask what the agent did on the 14th.
Back of the envelope
- full run record
- prompt, context, tools, diff, result
- model and prompt versions
- the fields that explain behaviour change
- retained to policy
- this is an audit trail
You did it
You just designed an agent harness.
Everything you assembled, in order
- The bottleneck moved from generation to verification — design for that, not for prompting.
- Capability comes only from tools: a narrow designed surface, never a shell.
- One ephemeral workspace per task makes failure cheap, which is what lets you run many agents.
- Decompose until each unit has a machine-checkable acceptance condition, or refuse the task.
- Verification must live outside the agent’s reach — if it can edit the check, there is no check.
- Route review by blast radius; that ratio, not the model, sets how many agents one engineer can supervise.
- Record prompt, model version, tool calls, diff and approver — agent work is production work.