The big idea
A test that always passes is worse than no test
Generating a test is trivial — an LLM can produce SOMETHING that looks like a test in seconds. Generating a test that actually catches real bugs, doesn't flake, and doesn't just lock in whatever the code currently does (bugs included) is the genuinely hard problem.
We'll build the whole validation pipeline a generated test needs to earn trust: coverage targeting, mutation testing to prove it's meaningful, flakiness checking, and a human review gate that catches what even mutation testing can't.
How to read this: Each step opens with a real design decision — make the call before I show you what ships. Watch the diagram grow, hover the boxes, and at the end kill mutation testing and generate a flaky test to see the safeguards catch what they're meant to. Hit Begin.
Step 1 · The baseline
Manual tests only
Simplest version: developers write all tests themselves, as part of normal development. What happens under real deadline pressure?
Design decision: Tests are written entirely manually, alongside normal feature work under deadline pressure. What tends to happen?
The call: Coverage gaps accumulate on new and changed code, especially under time pressure, leaving real paths through the code genuinely untested. — Manual testing effort is finite and competes with feature work — under pressure, coverage gaps are a predictable, common outcome, and those gaps are exactly where undetected bugs tend to live.
Manual testing effort competes directly with feature work, and under real deadline pressure, coverage gaps on new and changed code are a predictable, common outcome — exactly where undetected bugs tend to hide.
Testing effort is a finite resource competing with delivery pressure: This isn't a discipline problem to be solved by asking developers to try harder — it's a structural tension between testing thoroughness and delivery speed that automation can genuinely help resolve, if done well.
Step 2 · Find the gaps precisely
Target generation, don't duplicate coverage
Before generating anything, figure out exactly what's NOT already covered.
Design decision: Before generating tests, what should determine where generation effort goes?
The call: Analyze exactly which lines and branches of new/changed code currently have no test coverage, and target generation specifically at those gaps. — Precise coverage-gap analysis means every generated test adds genuinely NEW coverage rather than duplicating what already exists — the most efficient use of generation effort, focused exactly where the real risk (untested code paths) actually lives.
Add a Coverage-Gap Analyzer that identifies exactly which lines and branches of new or changed code lack test coverage, and targets generation precisely at those gaps — rather than generating redundant tests for already-covered code.
Target the actual gap, not blanket coverage: Precise gap-targeting is both more efficient (no wasted generation effort) and more effective (every new test adds real, previously-missing coverage) than a scattershot approach.
Step 3 · Generate against the real signature
A specific test, not generic boilerplate
Generate a test for an identified gap. Should it be a generic template, or something tailored to the actual code?
Design decision: Generating a test for a specific coverage gap — generic boilerplate, or something more targeted?
The call: A test generated against the function's actual signature and observed behavior, specifically exercising the identified uncovered path. — Generating directly against the real function — its actual parameters, return behavior, and the specific gap identified — produces a test that genuinely exercises the logic in question, not a generic template that happens to call the function without testing anything meaningful about it.
The Test Generator produces a test targeting the specific identified gap, built against the function's actual signature and behavior — exercising the real uncovered path, not a generic template that merely calls the function without meaningfully testing it.
Specificity is what makes a generated test worth having: A test's value comes from how precisely it exercises real logic — generating against the actual code's real interface and the specific identified gap is what separates a useful test from one that merely exists.
Step 4 · Prove the test is meaningful
Mutation testing: does it actually catch bugs?
A generated test passes. That alone doesn't prove much — a test that asserts nothing meaningful, or asserts something trivially true, also "passes." How do you verify a generated test actually catches real bugs?
Design decision: A generated test passes. How do you verify it's actually meaningful, not just trivially passing?
The call: Use MUTATION TESTING: deliberately introduce a small bug (a "mutant") into the code and confirm the generated test actually FAILS against that mutated, broken version — if the test still passes against clearly-broken code, it isn't testing anything meaningful and should be discarded or regenerated. — Mutation testing directly answers the question that matters: "would this test actually catch a real bug?" By deliberately breaking the code in a small way and checking the test fails, you get concrete proof the test is sensitive to the actual logic — not just passing by coincidence or triviality.
Validate every generated test with mutation testing: introduce a small, deliberate bug into the code and confirm the test fails against that mutant. A test that still passes against clearly-broken code isn't testing anything meaningful and is discarded or sent back for regeneration — this fails closed, so nothing unproven reaches the test suite.
Prove sensitivity to real bugs, don't just observe passing: This is a genuinely deep, underused testing technique in practice: mutation testing gives concrete, falsifiable evidence that a test would actually catch a real regression, which "it currently passes" alone can never prove.
Step 5 · Catch flakiness before it poisons CI
Inconsistent results are worse than no test
A test passed mutation testing — it genuinely exercises real logic. But does it produce the SAME result every time it runs?
Design decision: A generated test passed mutation testing. Is that sufficient to trust it in the CI suite?
The call: No — run the newly-validated test multiple times (or under varied conditions) to check for flakiness before trusting it in the suite; an inconsistent result is a real problem even for a test that's otherwise meaningful. — A flaky test — one that passes and fails inconsistently for reasons unrelated to actual code correctness — is arguably worse than no test at all, because it trains developers to distrust and ignore CI failures ("just rerun it"), which erodes confidence in the entire test suite, not just that one test.
Run every newly mutation-validated test through a Flakiness Checker — repeated executions to confirm consistent results — before it's trusted in the suite. A test that produces inconsistent results, even one that's otherwise meaningful, is discarded or flagged rather than allowed to poison CI signal.
A flaky test damages trust in the whole system, not just itself: Once developers learn that SOME failures are "just flaky, ignore it," that skepticism tends to bleed into how they treat every failure — flaky tests have an outsized negative effect on CI trust relative to their individual footprint.
Step 6 · Human review — for a different reason than mutation testing
Meaningful isn't the same as correct
A test passed mutation testing (it's sensitive to real bugs) AND the flakiness check (it's consistent). Is it now safe to auto-merge without any human involvement?
Route mutation-tested, non-flaky generated tests through a Human Review Gate before they join the suite — because mutation testing proves a test is non-trivial, not that its assertion reflects correct intended behavior. A test generated against code that's subtly buggy could pass mutation testing (it genuinely fails on further mutation) while still asserting the WRONG expected outcome, effectively locking in a bug as if it were correct behavior. A developer's review catches exactly this gap that automated validation structurally can't.
Two different questions, two different checks: "Is this test meaningful" (mutation testing answers this) and "does this test check the CORRECT thing" (only a human, or a spec/intent-anchored process, can answer this) are genuinely different questions — passing one doesn't imply passing the other.
Step 7 · Keep the fast lane fast
Expensive validation shouldn't block every push
Generation, mutation testing, and flakiness checking are all relatively expensive and slow. Should they run in the same fast lane developers wait on for every push?
Run test generation and its validation pipeline in a separate, asynchronous lane — the fast primary CI feedback loop developers wait on for every push stays fast and unaffected, while generation and mutation/flakiness validation happen in the background, with results (a proposed new test, pending review) surfacing once ready rather than blocking anything in the interactive path.
Separate latency-sensitive from latency-tolerant work: The same principle from other designs' latency budgeting: work that must be fast (the primary CI signal developers actively wait on) and work that can tolerate delay (test generation and its expensive validation) belong in different lanes, not competing for the same time budget.
Step 8 · The sharp edges
Don't lock in a bug, and prefer testing behavior over implementation
Two subtler risks: generating a test against code that's CURRENTLY buggy would validate the bug as expected behavior, and tests tightly coupled to internal implementation details break on every refactor even when the actual behavior is unchanged.
Anchor generation to some notion of intent — a docstring, a spec, a PR description — rather than purely "whatever the current code happens to do," so a generated test doesn't simply codify an existing bug as if it were correct. And prefer generating tests against a function or module's public behavior and interface rather than internal implementation details, so tests remain valid across refactors that preserve behavior but change internals — reducing long-term maintenance burden on the generated suite.
Design for the unhappy path: Mutation testing down → fail closed, nothing unproven merges. A flaky result → caught before it reaches the suite. Buggy code → generation anchored to intent, not just current behavior. Implementation churn → test against public behavior, not internals. A test-generation agent that only works for already-correct code with a stable implementation is a demo; one that resists locking in bugs and survives refactors is a product.
You did it
You just designed a CI test-generation agent.
- Manual testing effort competes with feature delivery, and coverage gaps are a predictable outcome under real deadline pressure.
- Coverage-gap analysis targets generation precisely at what's genuinely uncovered, not redundantly at what's already tested.
- Tests are generated against the real function signature and behavior, exercising the specific identified gap.
- Mutation testing proves a generated test is meaningful — actually sensitive to real bugs — not just trivially passing.
- A flakiness check catches inconsistent test results before they poison CI trust, which a flaky test does disproportionately.
- A human review gate catches what mutation testing structurally can't: whether a test's assertion reflects genuinely correct intended behavior, not just non-trivial behavior.
- Generation and its expensive validation run in an async lane, keeping the fast primary CI feedback loop unaffected.
- Generation anchored to intent (not just current behavior) avoids locking in existing bugs, and testing public behavior over internals reduces refactor-driven maintenance burden.