The whole design, in writing
Every step of the build above, written out: the problem each piece solves, the option that was taken and the ones that were not, the numbers, and how it fails in production.
The big idea
A test that always passes is worse than no test
Generating a test is trivial — an LLM can produce SOMETHING that looks like a test in seconds. Generating a test that actually catches real bugs, doesn't flake, and doesn't just lock in whatever the code currently does (bugs included) is the genuinely hard problem.
We'll build the whole validation pipeline a generated test needs to earn trust: coverage targeting, mutation testing to prove it's meaningful, flakiness checking, and a human review gate that catches what even mutation testing can't.
What the new pieces do
- Developerclient
- Ships code under real deadline pressure, where comprehensive manual test-writing is often the first thing to get shortchanged.
Step 1 · The baseline
Manual tests only
Simplest version: developers write all tests themselves, as part of normal development. What happens under real deadline pressure?
Tests are written entirely manually, alongside normal feature work under deadline pressure. What tends to happen?
In practice, thorough test-writing is one of the first things to get shortchanged under real deadline pressure — not because developers don't value it, but because it competes directly with shipping the feature itself, and shipping usually wins under a deadline.
Manual testing effort is finite and competes with feature work — under pressure, coverage gaps are a predictable, common outcome, and those gaps are exactly where undetected bugs tend to live.
Coverage gaps from time pressure occur at any codebase size — the underlying dynamic (testing effort competing with feature work under deadline pressure) doesn't depend on scale.
Manual testing effort competes directly with feature work, and under real deadline pressure, coverage gaps on new and changed code are a predictable, common outcome — exactly where undetected bugs tend to hide.
Why this piece earns its place
The baseline step is also the deployment decision, and it is easy to skim past: everything downstream lives inside CI rather than in an editor plugin or a nightly cron. A generated test is worth nothing until it has been executed, and executing it needs a checked-out tree, installed dependencies and a working test command pinned to one commit. CI is the only place that already has all three, reproducibly, for the exact code under discussion — rebuilding that environment anywhere else is most of the work of building CI again. What that binds you to, though, is a toolchain rather than a codebase, and that is the second-order cost nobody prices in. Coverage instrumentation, the test runner, the assertion conventions and later the mutation operators are every one of them language-specific. A repo with a Python service behind a TypeScript front end is two builds of this agent, not one with a flag, and the second language costs close to what the first did. Work that out before you start, because it usually decides that the honest scope is the one stack carrying most of the risk rather than the whole repo. Where you would skip the design entirely: a repo with no CI. There the first deliverable is CI, and this conversation is a quarter away.
What the new pieces do
- CI Pipelinegateway
- Runs on every push — the fast feedback loop developers wait on, which nothing in this design should be allowed to slow down significantly.
Step 2 · Find the gaps precisely
Target generation, don't duplicate coverage
Before generating anything, figure out exactly what's NOT already covered.
Before generating tests, what should determine where generation effort goes?
Generating redundant tests for already well-covered code wastes generation effort and adds maintenance burden for no real coverage benefit — the goal is filling GAPS, not maximizing raw test count.
Precise coverage-gap analysis means every generated test adds genuinely NEW coverage rather than duplicating what already exists — the most efficient use of generation effort, focused exactly where the real risk (untested code paths) actually lives.
Relying on developers to manually flag which functions need generated tests reintroduces the same time-pressure problem from Step 1 — remembering to flag things under deadline pressure is exactly the kind of extra step that gets skipped.
Add a Coverage-Gap Analyzer that identifies exactly which lines and branches of new or changed code lack test coverage, and targets generation precisely at those gaps — rather than generating redundant tests for already-covered code.
Why this piece earns its place
Gap analysis is the component whose degradation is invisible from the outside. If the coverage report comes back empty — an instrumentation flag dropped during a build refactor, a runner upgraded, a language the tool cannot parse — the analyzer finds no gaps, the generator proposes nothing, and every dashboard stays green. Nothing fails. The agent just quietly stops working, and someone notices weeks later that no proposals have arrived. So the alarm that matters is on throughput, not errors: gaps found per merged change, proposals produced per day. Two details worth naming out loud. Branch coverage and line coverage disagree, and the disagreement is where the interesting gaps hide — a line inside a conditional can be fully covered while one side of that condition never runs once. And the definition of changed code is a per-request knob rather than a config constant: the diff against the merge base is what a reviewer cares about, the diff against the previous push is what a developer iterating on a branch cares about, and fixing one globally serves the other badly.
What the new pieces do
- Coverage-Gap Analyzerservice
- Identifies exactly which lines and branches of new/changed code have no test coverage — so generation effort is spent precisely, not redundantly.
Step 3 · Generate against the real signature
A specific test, not generic boilerplate
Generate a test for an identified gap. Should it be a generic template, or something tailored to the actual code?
Generating a test for a specific coverage gap — generic boilerplate, or something more targeted?
Generic boilerplate rarely exercises the SPECIFIC logic of the actual uncovered code path — it might technically execute the function without meaningfully testing its real behavior or edge cases.
Generating directly against the real function — its actual parameters, return behavior, and the specific gap identified — produces a test that genuinely exercises the logic in question, not a generic template that happens to call the function without testing anything meaningful about it.
Merely checking for the absence of an exception is a very weak test — it wouldn't catch a function that runs without crashing but produces the WRONG result, which is a common and important class of bug.
The Test Generator produces a test targeting the specific identified gap, built against the function's actual signature and behavior — exercising the real uncovered path, not a generic template that merely calls the function without meaningfully testing it.
Why this piece earns its place
The signature is the smallest part of what the generator needs. It also needs the module's imports, the fixtures the repo already has, the framework and assertion style in use, and where tests for this module live. Skip that and you get tests that are individually reasonable and collectively foreign — a second set of helpers, a second mocking style, a second way to build the same object. Nobody rejects those at review — each looks fine alone — and the suite grows a dialect only the agent speaks. Two failure counters are worth keeping apart, because they point at different fixes and only one is legible here. A candidate that does not import or compile fails immediately and locally: the generator was not shown enough of the repo, and context selection changes on the next run. A candidate that compiles, runs and asserts nothing looks exactly like a good one from here — nothing at generation time tells them apart, which is the job Step 4 exists to do. So attribute that verdict backwards: a mutant left alive is a fact about the generator, not only about the test it came back on, and filing it purely as a validation-stage rejection loses the one number saying the prompt or the model needs changing. Collapsing both into one bad-generation rate hides which lever to pull, and the levers cost wildly different amounts.
What the new pieces do
- Test Generatorservice
- Generates a test exercising the specific uncovered path, using the actual function's real signature and behavior.
Step 4 · Prove the test is meaningful
Mutation testing: does it actually catch bugs?
A generated test passes. That alone doesn't prove much — a test that asserts nothing meaningful, or asserts something trivially true, also "passes." How do you verify a generated test actually catches real bugs?
A generated test passes. How do you verify it's actually meaningful, not just trivially passing?
A test can pass for the wrong reason entirely — asserting something that's always true regardless of the code's actual behavior, or not really exercising the logic at all — "passes" alone says nothing about whether the test would actually catch a real bug.
Mutation testing directly answers the question that matters: "would this test actually catch a real bug?" By deliberately breaking the code in a small way and checking the test fails, you get concrete proof the test is sensitive to the actual logic — not just passing by coincidence or triviality.
A second model's subjective assessment of whether a test "looks right" doesn't provide the same concrete, verifiable proof that mutation testing does — mutation testing gives an objective, falsifiable answer (does the test actually fail on broken code), not a judgment call.
Validate every generated test with mutation testing: introduce a small, deliberate bug into the code and confirm the test fails against that mutant. A test that still passes against clearly-broken code isn't testing anything meaningful and is discarded or sent back for regeneration — this fails closed, so nothing unproven reaches the test suite.
Why this piece earns its place
One ordering mistake makes the whole check worthless. Run the candidate against the unmutated code first and require it to pass. A test that fails to import, or that was simply broken on arrival, also fails against the mutant, and a naive check reads that as proof of sensitivity. The two runs are halves of one check, not alternatives. The operator set is the other thing to choose deliberately, because it is the closest this design comes to writing down what meaningful means. Deleting a statement is caught by nearly any test that touches the line; flipping a comparison or moving a boundary is caught only by one that checks the result. Tighten the operators later and you have silently raised the bar on everything admitted before, so store the standard a test passed beside the test. Two more things you only learn by running it. Equivalent mutants — edits that change no behavior — can never be killed, so counting them as rejections discards good tests. And a kill is a single execution, taken before Step 5 has established the candidate is even stable, so record which mutant died: that is what re-proves the verdict if the test later turns out to be unreliable.
- mutant introduceda deliberate small bug
- test must FAILagainst the mutant to prove value
- fails closedunproven tests never merge
What the new pieces do
- Mutation Testerservice
- Introduces a small deliberate bug into the code and confirms the generated test actually fails against it — proof the test is meaningful, not trivially passing.
Step 5 · Catch flakiness before it poisons CI
Inconsistent results are worse than no test
A test passed mutation testing — it genuinely exercises real logic. But does it produce the SAME result every time it runs?
A generated test passed mutation testing. Is that sufficient to trust it in the CI suite?
Mutation testing proves the test is sensitive to real bugs, but says nothing about whether it produces CONSISTENT results — a test could be meaningful and still be flaky (timing-dependent, order-dependent), which is a separate failure mode entirely.
A flaky test — one that passes and fails inconsistently for reasons unrelated to actual code correctness — is arguably worse than no test at all, because it trains developers to distrust and ignore CI failures ("just rerun it"), which erodes confidence in the entire test suite, not just that one test.
Generated tests can be just as prone to flakiness (timing assumptions, shared state, ordering dependencies) as manually-written ones — there's no reason to assume generation exempts a test from this risk.
Run every newly mutation-validated test through a Flakiness Checker — repeated executions to confirm consistent results — before it's trusted in the suite. A test that produces inconsistent results, even one that's otherwise meaningful, is discarded or flagged rather than allowed to poison CI signal.
Why this piece earns its place
Repeated runs are a filter, not a proof. Anything that misbehaves rarely enough walks through whatever batch you can afford and surfaces much later, in an unrelated pull request, as a red build nobody can explain. That is the consequence worth planning for: this check needs a partner inside the suite — a quarantine path that can demote a merged test which starts misbehaving, and attribute it back to the run that produced it. Without that edge the checker never learns what it let past, and its run budget stays a guess. How you spend that budget matters more than how large it is. Most instability comes from shared state, contention, and the order the suite happens to execute in, none of which shows up when one test runs alone, over and over, on an idle machine. Varying the conditions finds more than repeating them: shuffle the order, run it alongside the rest of the suite, run it under parallelism. And handle one case carefully — sometimes the test is fine and the code is racy. Then the inconsistency is a real defect this agent just found, and a silent discard throws away the most valuable thing it has produced. Discard by default, but never into a counter nobody reads.
What the new pieces do
- Flakiness Checkerservice
- Runs a newly-validated test multiple times before trusting it — a test with inconsistent results is worse than no test at all.
Step 6 · Human review — for a different reason than mutation testing
Meaningful isn't the same as correct
A test passed mutation testing (it's sensitive to real bugs) AND the flakiness check (it's consistent). Is it now safe to auto-merge without any human involvement?
Route mutation-tested, non-flaky generated tests through a Human Review Gate before they join the suite — because mutation testing proves a test is non-trivial, not that its assertion reflects correct intended behavior. A test generated against code that's subtly buggy could pass mutation testing (it genuinely fails on further mutation) while still asserting the WRONG expected outcome, effectively locking in a bug as if it were correct behavior. A developer's review catches exactly this gap that automated validation structurally can't.
Why this piece earns its place
A review gate is a queue, and queues that grow get rubber-stamped, which converts the gate into a formality that still charges full price in delay. So the design question is not whether to review, it is what one review costs. Route each proposal to the author of the code it covers, while the intent is still in their head, and keep the unit small — one test, one decision, not a batch handed over at the end of a sprint. Show them the mutant that test killed beside the diff: that is evidence they cannot cheaply reconstruct, and it moves the question from is this a good test, which invites a slow reread of the whole function, to is this the behavior I meant, which is the only thing a human is here for. What teams miss is that this gate is also the admission policy for the suite. An accepted test is a permanent cost — it runs on every future push, and from then on every change touching that code has to satisfy it or argue with it. So the reviewer is really answering two questions, and the second one, is this worth paying for indefinitely, is what keeps a generated suite a signal rather than a tax. Which means the reject has to stay cheap and unembarrassing to use, or the first question quietly becomes the only one.
What the new pieces do
- Human Review Gateservice
- A developer sanity-checks that a generated, mutation-proven test actually reflects CORRECT intended behavior, not just that it isn't trivial.
- Test Suitedata
- The durable, trusted set of tests that actually gates future changes — every entry has survived generation, mutation testing, flakiness checking, and human review.
Step 7 · Keep the fast lane fast
Expensive validation shouldn't block every push
Generation, mutation testing, and flakiness checking are all relatively expensive and slow. Should they run in the same fast lane developers wait on for every push?
Run test generation and its validation pipeline in a separate, asynchronous lane — the fast primary CI feedback loop developers wait on for every push stays fast and unaffected, while generation and mutation/flakiness validation happen in the background, with results (a proposed new test, pending review) surfacing once ready rather than blocking anything in the interactive path.
Why this piece earns its place
Moving work off the critical path buys latency and costs freshness. By the time a proposal is ready the branch has usually moved, and a test written against code that has since been rewritten spends a reviewer on a decision that no longer applies — which is how a team learns to skim every proposal that follows. So a proposal carries the commit it was generated against, and on arrival the lane checks whether the targeted lines still exist in that form: revalidate, or discard silently. Never surface a stale one. Two consequences are easy to miss. This lane has no verdict to give the push that triggered it, so its output cannot be a CI status check at all — a proposal is a change, and it needs somewhere to land as one, which is also what gives Step 6's reviewer something to approve or close. And separate lane is a scheduling claim, not a topology one. Where both lanes draw executors from the same runner pool, a burst of pushes puts generation work in front of the loop you promised not to slow, so give it its own capacity or a strictly lower priority class, and bound the backlog rather than let it grow.
Step 8 · The sharp edges
Don't lock in a bug, and prefer testing behavior over implementation
Two subtler risks: generating a test against code that's CURRENTLY buggy would validate the bug as expected behavior, and tests tightly coupled to internal implementation details break on every refactor even when the actual behavior is unchanged.
Anchor generation to some notion of intent — a docstring, a spec, a PR description — rather than purely "whatever the current code happens to do," so a generated test doesn't simply codify an existing bug as if it were correct. And prefer generating tests against a function or module's public behavior and interface rather than internal implementation details, so tests remain valid across refactors that preserve behavior but change internals — reducing long-term maintenance burden on the generated suite.
Why this piece earns its place
Intent sources are not interchangeable, and ranking them is worth doing out loud. A ticket or a spec is written before the code and usually by someone other than its author, so it can genuinely contradict the implementation — which is exactly what makes it useful. A docstring sits next to the code and rots with it. A pull request description is written by the same person who just wrote the bug, describing what they believe they built: closest to the change, weakest as an independent check. The useful behavior when intent and code disagree is not to generate a test at all. It is to say so. An agent that flags the contradiction is worth more than one that quietly picks a side, and picking the code is how a bug gets frozen into the suite as a requirement. The public-versus-internal rule needs something computable behind it too — exported symbols, a package boundary, whatever the language actually enforces — because where nothing enforces a boundary, every function looks public. And mark generated tests as generated. It is one annotation, and it is what makes them attributable later and cheap to regenerate in bulk once the generator improves.
You did it
You just designed a CI test-generation agent.
Everything you assembled, in order
- Manual testing effort competes with feature delivery, and coverage gaps are a predictable outcome under real deadline pressure.
- Coverage-gap analysis targets generation precisely at what's genuinely uncovered, not redundantly at what's already tested.
- Tests are generated against the real function signature and behavior, exercising the specific identified gap.
- Mutation testing proves a generated test is meaningful — actually sensitive to real bugs — not just trivially passing.
- A flakiness check catches inconsistent test results before they poison CI trust, which a flaky test does disproportionately.
- A human review gate catches what mutation testing structurally can't: whether a test's assertion reflects genuinely correct intended behavior, not just non-trivial behavior.
- Generation and its expensive validation run in an async lane, keeping the fast primary CI feedback loop unaffected.
- Generation anchored to intent (not just current behavior) avoids locking in existing bugs, and testing public behavior over internals reduces refactor-driven maintenance burden.