The whole design, in writing
Every step of the build above, written out: the problem each piece solves, the option that was taken and the ones that were not, the numbers, and how it fails in production.
The big idea
What is an enterprise eval pipeline?
A customer says the system “should be accurate”. That sentence is not a specification — it is the opening of a negotiation you will have again in six months, under worse conditions, when they are unhappy and you are holding a number nobody agreed to.
Build the measurement before you build the promise: a frozen golden set, a ceiling derived from how often the customer’s own experts agree with each other, a calibrated judge to grade at volume, a CI gate that blocks merges, and a drift monitor so the whole thing stays true after launch.
Step 1 · The skeleton
Run the system on some cases
The naive version: an engineer changes a prompt, tries five examples by hand, decides it looks better, and merges. It works until the change that improved those five broke forty others — and nobody finds out until the customer does.
Stand up an Eval Runner that executes the system against a fixed set of cases and records what happened. Everything after this makes those runs comparable, trustworthy, and consequential.
Why this piece earns its place
The first thing this runner will tell you is that the system does not give the same answer twice. Sampling, retrieval order, a provider that quietly reroutes you — run the same cases against an unchanged system and a handful of results flip. That variance is the noise floor, and until somebody measures it every difference in score is ambiguous: you cannot tell a real improvement from a rerun. So before anything depends on these numbers, pin what can be pinned — sampling down, seeds where they exist, the retrieval index frozen — then run the unchanged system against the set repeatedly and record how far the overall score and the per-case results move. That band is what the later steps are allowed to call a change. It also fixes the honest resolution of the whole pipeline: if an unchanged system swings further between runs than the improvement someone is claiming, no threshold can defend that claim, and the fix is more cases or less sampling rather than a stricter rule.
What the new pieces do
- Engineerclient
- Anyone changing the system — a prompt edit, a retrieval tweak, a model upgrade. Every one of those is a measurement change, and none of them feels like one at the time.
- Eval Runnerbackend
- Executes the system under test against every case, deterministically, recording the output, the retrieved sources, the model version, the prompt version, latency and cost per case.
Step 2 · The set
Golden set: real, stratified, frozen
If the cases change between runs, no two numbers can be compared. And if you sample them uniformly, the rare high-cost cases — the ones that actually cost the customer money — barely appear, so the score is dominated by the easy majority.
You have 9,000 real cases and want a golden set. What do you build?
Random sampling under-represents the rare, expensive cases, and refreshing it means no two runs are comparable. You would have built a number that moves for reasons nobody can attribute.
Cost and latency per run make this unusable in CI, and — more importantly — most of those cases have never been checked by anyone who knew the right answer. Volume is not the constraint; label quality is.
Small enough to run on every PR, stratified so the expensive tail is visible, and frozen so scores mean the same thing next quarter. Every case checked by someone who knew what they were doing.
Build a Golden Set of 100–300 real cases, stratified so rare high-cost classes are over-represented, frozen under version control, with every case labelled by someone who knew the answer.
Why this piece earns its place
Stratification is the right call, and it quietly breaks the headline number. Once the rare high-cost classes are over-represented, the overall score stops being an estimate of what the customer will experience in production — it is the score on a set deliberately built to be harder than reality. That is fine right up until someone reads it as an accuracy claim, which someone always does, usually in a slide. So keep the strata visible: report each class as its own number, and when a production-like figure is genuinely needed, reweight the classes back to their real frequencies rather than averaging the set as it stands. The tilt is a knob with a cost in both directions. Too little and the expensive tail is averaged away by the easy majority, which is the failure this step exists to fix. Too much and the rare class dominates a score that barely moves for real users — at which point engineers learn the number is not about them and stop reading it. Tilt until the expensive class can visibly move the result, then stop.
- 100–300cases
- stratifiednot uniform
- frozenversion-controlled
What the new pieces do
- Golden Setdata
- Real, agreed-correct cases under version control, stratified so rare high-cost cases are over-represented. Typically 100–300. Changing it is a decision with a reviewer, not a fix for a failing test.
Back of the envelope
- real cases
- from work the customer already did
- stratified
- rare high-cost classes over-represented
- frozen
- edits are reviewed decisions
Step 3 · The ceiling
Measure how often two experts agree
Someone in a sales meeting said 95%. Nobody knows whether 95% is easy, hard, or impossible for this task — because nobody has measured how often two humans doing the job give the same answer. Without that, every later conversation about quality is an argument between opinions.
Before promising an accuracy target, what is the single cheapest thing to measure?
Public benchmarks measure a different distribution with different labelling conventions. They tell you nothing about whether 95% is achievable on this customer’s cases, which is the only question that matters here.
One afternoon of work, and it establishes the practical ceiling. It also produces something more valuable: the cases they disagree on are the rules nobody had written down.
Useful for the ROI story later, and it says nothing about achievable quality. You would still be promising a number with no evidence that it is reachable.
Have two experts label the same 40 cases independently, then measure agreement. That rate is your ceiling, and the targets go into the contract relative to it: “match or exceed the 72% expert baseline, with under 2% in the high-cost error class”.
Why this piece earns its place
The cheaper version of this is one senior expert writing the answer key alone, and it loses for a reason that is easy to miss. One labeller produces labels but no ceiling — and no way to separate a genuinely ambiguous case from a clear one. Afterwards every case the system failed looks identical: wrong. Two people labelling the same cases independently produce a third category, the cases that experienced practitioners genuinely split on, and that category is where most of the later argument about quality actually lives. A second labeller also protects you from the specific failure of one person having house rules nobody else follows, which turns the whole exercise into a measurement of that person rather than of the task. The cost is real — two experts’ time on the same cases instead of twice the coverage — but it is an afternoon, and it is the only version that produces a number you can defend in a contract instead of an opinion you have to keep re-arguing.
- 40 casesone afternoon
- 2 expertsindependently
- the ceilingnot 100%
What the new pieces do
- Human Ceilingservice
- Two of the customer’s own experts label the same 40 cases independently. The rate at which they agree is the practical ceiling — and the number every later quality argument rests on.
Back of the envelope
- target relative to the human baseline
- not a round number
- separate cap on high-cost errors
- costs are asymmetric
- disagreements resolved
- become the written rule
Step 4 · Grade at volume
An LLM judge, reported with its calibration
Humans cannot grade 300 cases on every pull request. An LLM judge can — and an uncalibrated judge produces confident numbers that nobody should act on, because you cannot tell a real regression from a judge quirk.
Add an LLM Judge that receives the rubric verbatim, is blind to which system produced the output, uses a small discrete scale, and is pinned to a model version. Then grade the same cases with humans and report the agreement rate beside every score.
Why this piece earns its place
Agreement is not a property the judge has once. It is measured on outputs from the system as it was on the day you measured it, and the system under test is precisely the thing that keeps changing — a rewritten prompt moves the outputs into shapes the calibration sample never contained. So treat the figure as perishable: re-measure it against a fresh human-graded sample on a cadence, and always after a judge model change or a substantial change to the system being graded. The second rule is to refuse to report it as a single number. Judges tend to agree with humans almost perfectly on the easy majority and worst on exactly the rare, contested, expensive cases, so a healthy overall agreement rate can sit on top of a judge that is close to guessing where it matters most. Report agreement per case class, because the high-cost class is the one the gate’s error budget rests on, and that is the number that has to be trustworthy.
- rubric verbatimnot paraphrased
- blindto system identity
- pinned versionupgrades are changes
What the new pieces do
- LLM Judgeservice
- Grades at volume against the rubric verbatim, blind to which system produced the output, pinned to a model version. Its agreement with human grades is reported beside every score it emits.
Back of the envelope
- judge agreement reported
- beside every score
- blind and rubric-faithful
- small discrete scale
- version pinned
- a judge upgrade is a re-baseline
Step 5 · Make it a control
The gate that blocks the merge
Everything so far produces a number. A number in a dashboard changes nothing — the prompt still gets edited on a Friday, the model still gets upgraded, and the regression still ships.
Which assertion catches the failure an overall average hides?
Necessary and insufficient. The average can rise while the rare expensive cases break — which is precisely the failure the customer notices first.
Both belong in the gate, and neither is about correctness. A fast, cheap, wrong system passes this assertion comfortably.
Per-case regression tracking. It is the assertion that earns its keep, because it detects the specific silent breakages that any aggregate metric averages away.
Add a CI Gate backed by a Results Store of per-case history. It asserts the overall score, a high-cost error budget, an empty regression list, and the latency and cost ceilings — on every PR, including every prompt change.
Why this piece earns its place
The interesting failure of a gate is social rather than technical. The first time it blocks something a customer is waiting for, someone asks how to turn it off, and whatever answer exists in that moment becomes the real policy. If there is no override, it gets commented out of the CI config at six in the evening and nobody turns it back on. If the override is one click and leaves no record, the gate is advisory again within a quarter. So design the override deliberately: it exists, it requires a named approver, it records which cases were failing and why shipping anyway was acceptable, and it expires — the failing cases come back as work instead of silently becoming the new baseline. The record is also the most useful signal this whole pipeline produces. Repeated overrides on the same case class rarely mean the team is undisciplined; they usually mean the budget was set against cases the business does not actually consider expensive, and it is the budget that should move.
- every PRincl. prompt edits
- regressionsmust be empty
- error budgethigh-cost class
What the new pieces do
- Results Storedata
- Every run, every case, every version. This is what makes "which cases used to pass and now fail?" answerable — the question that catches the failures an average hides.
- CI Gatecache
- Asserts overall score against the human baseline, a high-cost error budget, an empty regression list, and the latency and cost ceilings. Until an eval blocks a merge it is a document, not a control.
Back of the envelope
- overall ≥ human baseline
- agreed in the contract
- high-cost error rate ≤ budget
- asymmetric costs
- zero per-case regressions
- the assertion that saves you
- p95 latency + cost per case
- quality is not only correctness
Step 6 · Stay true
Drift: the world moves away from your test set
The golden set is a snapshot. Six months later the customer’s document mix has changed, a new vendor sends a different format, and the model provider shipped an update. The gate is still green and the users are still unhappy.
Sample production continuously and score it. Alert separately on input-distribution drift (the cases changed) and score drift (the answers got worse) — different causes, different fixes. Promote new failure classes back into the golden set as a reviewed decision.
Why this piece earns its place
There is one thing production cannot hand you, and it is the answer key. On the golden set a score compares an output against something a person agreed was correct; on live traffic there is nothing to compare against, so scoring a sample means asking the judge to rate an answer against the rubric with no reference — a weaker measurement that inherits every calibration caveat the judge already carries. That is why the two alerts deserve to be trusted differently, not just separated. Input-distribution drift needs no labels at all: it compares this week’s incoming cases against the production mix recorded at launch — the real class frequencies the golden set is deliberately tilted away from, the same ones you keep in order to reweight a score back to production-like. Baseline it on the golden set’s own stratified shape instead and the first run reports a designed-in gap that never clears. Measured against the launch mix it is cheap, it is solid, and it usually fires first, because the cases change before the answers get worse. Score drift on unlabelled traffic is the softer of the two, and I would treat it as a prompt to go and get real labels on a sample rather than as something to page on directly — which is what makes the one-click correction control in the product a monitoring component, not a nicety.
- sampledcontinuously
- two alertsdistribution vs score
- new classespromoted to golden
What the new pieces do
- Productionclient
- The customer’s live workload. It is the only source of the cases you did not think of, and the distribution it produces moves away from your test set continuously.
- Drift Monitorservice
- Scores a continuous sample of live traffic, alerting separately on input-distribution drift and on score drift — they have different causes and different fixes.
Back of the envelope
- production sampling
- the only source of unknown cases
- distribution drift ≠ score drift
- separate alerts
- user correction control
- labels for free
Step 7 · The number that decides
The business view
Your dashboards show F1, judge agreement and p95 latency. The person deciding the renewal does not know what any of those are, and will make the decision anyway — on whatever they can remember.
Publish one agreed business metric on a fixed cadence, traceable back to eval runs and adoption telemetry rather than to anecdote. Exception-resolution rate. Hours returned. Deflection rate. Whatever they agreed to in step 3.
Why this piece earns its place
The caveat worth saying out loud the first time this number is presented is that it moves for reasons the system does not own. Adoption climbs, a team reorganises, a seasonal peak arrives, a process upstream changes — and the metric moves without anything changing in the model. If the first time that is discussed is the quarter the number falls, it reads as an excuse. So agree the definition in writing before the first report: what is in the numerator, what is in the denominator, which cases are excluded, and what the comparison is — the period before launch, or a group still working the old way. Almost every fight about a business metric turns out to be a fight about the denominator, and it is far cheaper to have that one while everybody is still optimistic. Then keep attribution and correlation separate in the report itself: here is the number, here is the part traceable to eval runs, and here is the part that is not mine to claim.
- one numberthey agreed to
- fixed cadencenot at renewal
- traceableto eval runs
What the new pieces do
- Business Viewdata
- One metric the customer agreed to, on a cadence, traceable to eval runs and adoption telemetry. This is the artefact the renewal conversation actually turns on.
Back of the envelope
- one agreed business metric
- in their language
- delivered on a cadence
- a first review at renewal reads as a sales meeting
- traceable to runs
- evidence, not assertion
You did it
You just designed an enterprise eval pipeline.
Everything you assembled, in order
- Measure the human ceiling first — the target is relative to expert agreement, never a round number.
- Golden set: real, stratified toward high-cost cases, frozen, edited only by reviewed decision.
- A judge without a reported human-agreement rate is an instrument without a calibration.
- The gate must assert per-case regressions, not just an average — averages hide the expensive failures.
- Sample production continuously; alert separately on distribution drift and score drift.
- Report one agreed business metric on a cadence, traceable to the runs behind it.