The big idea
What is an enterprise eval pipeline?
A customer says the system “should be accurate”. That sentence is not a specification — it is the opening of a negotiation you will have again in six months, under worse conditions, when they are unhappy and you are holding a number nobody agreed to.
Build the measurement before you build the promise: a frozen golden set, a ceiling derived from how often the customer’s own experts agree with each other, a calibrated judge to grade at volume, a CI gate that blocks merges, and a drift monitor so the whole thing stays true after launch.
How to read this: We add one piece at a time, problem then fix, and the diagram grows. Hit Begin.
Step 1 · The skeleton
Run the system on some cases
The naive version: an engineer changes a prompt, tries five examples by hand, decides it looks better, and merges. It works until the change that improved those five broke forty others — and nobody finds out until the customer does.
Stand up an Eval Runner that executes the system against a fixed set of cases and records what happened. Everything after this makes those runs comparable, trustworthy, and consequential.
Eyeballing is not measurement: Five examples chosen by the person who made the change is the least reliable evaluation available, because it samples exactly the cases they were already thinking about.
Step 2 · The set
Golden set: real, stratified, frozen
If the cases change between runs, no two numbers can be compared. And if you sample them uniformly, the rare high-cost cases — the ones that actually cost the customer money — barely appear, so the score is dominated by the easy majority.
Design decision: You have 9,000 real cases and want a golden set. What do you build?
The call: 100–300 stratified cases, over-representing rare high-cost classes, frozen under version control. — Small enough to run on every PR, stratified so the expensive tail is visible, and frozen so scores mean the same thing next quarter. Every case checked by someone who knew what they were doing.
Build a Golden Set of 100–300 real cases, stratified so rare high-cost classes are over-represented, frozen under version control, with every case labelled by someone who knew the answer.
Freeze it, and treat edits as decisions: The most common way an eval pipeline dies is a case being quietly edited because it was failing. Changing the set is a reviewed decision with a recorded reason — the same bar as a schema migration.
Step 3 · The ceiling
Measure how often two experts agree
Someone in a sales meeting said 95%. Nobody knows whether 95% is easy, hard, or impossible for this task — because nobody has measured how often two humans doing the job give the same answer. Without that, every later conversation about quality is an argument between opinions.
Design decision: Before promising an accuracy target, what is the single cheapest thing to measure?
The call: How often two of the customer’s own experts agree with each other on the same 40 cases. — One afternoon of work, and it establishes the practical ceiling. It also produces something more valuable: the cases they disagree on are the rules nobody had written down.
Have two experts label the same 40 cases independently, then measure agreement. That rate is your ceiling, and the targets go into the contract relative to it: “match or exceed the 72% expert baseline, with under 2% in the high-cost error class”.
The disagreements are the specification: Put the experts in a room with the cases they split on. The argument they have is the business rule nobody documented — and it becomes the labelling guide, the judge rubric, and the acceptance criteria at once.
Step 4 · Grade at volume
An LLM judge, reported with its calibration
Humans cannot grade 300 cases on every pull request. An LLM judge can — and an uncalibrated judge produces confident numbers that nobody should act on, because you cannot tell a real regression from a judge quirk.
Add an LLM Judge that receives the rubric verbatim, is blind to which system produced the output, uses a small discrete scale, and is pinned to a model version. Then grade the same cases with humans and report the agreement rate beside every score.
A judge is a measuring instrument: An instrument without a calibration figure is decoration. Publish judge-versus-human agreement next to the score, inspect every disagreement, and treat a judge model upgrade as a measurement change — because it is one.
Step 5 · Make it a control
The gate that blocks the merge
Everything so far produces a number. A number in a dashboard changes nothing — the prompt still gets edited on a Friday, the model still gets upgraded, and the regression still ships.
Design decision: Which assertion catches the failure an overall average hides?
The call: The list of cases that used to pass and now fail is empty. — Per-case regression tracking. It is the assertion that earns its keep, because it detects the specific silent breakages that any aggregate metric averages away.
Add a CI Gate backed by a Results Store of per-case history. It asserts the overall score, a high-cost error budget, an empty regression list, and the latency and cost ceilings — on every PR, including every prompt change.
Until it blocks a merge it is a document: The transition from "we have evals" to "evals are a control" happens exactly when someone’s pull request is refused. Everything before that is reporting.
Step 6 · Stay true
Drift: the world moves away from your test set
The golden set is a snapshot. Six months later the customer’s document mix has changed, a new vendor sends a different format, and the model provider shipped an update. The gate is still green and the users are still unhappy.
Sample production continuously and score it. Alert separately on input-distribution drift (the cases changed) and score drift (the answers got worse) — different causes, different fixes. Promote new failure classes back into the golden set as a reviewed decision.
Build the "this was wrong" button on day one: A one-click correction control in the product turns six weeks of real usage into labelled data. It is the cheapest labelling pipeline available and almost nobody builds it before they need it.
Step 7 · The number that decides
The business view
Your dashboards show F1, judge agreement and p95 latency. The person deciding the renewal does not know what any of those are, and will make the decision anyway — on whatever they can remember.
Publish one agreed business metric on a fixed cadence, traceable back to eval runs and adoption telemetry rather than to anecdote. Exception-resolution rate. Hours returned. Deflection rate. Whatever they agreed to in step 3.
Technical metrics inform, business metrics decide: Keep both, report the one they agreed to, and make the trace from that number back to the eval runs inspectable — so when it is challenged, the answer is evidence rather than assertion.
You did it
You just designed an enterprise eval pipeline.
- Measure the human ceiling first — the target is relative to expert agreement, never a round number.
- Golden set: real, stratified toward high-cost cases, frozen, edited only by reviewed decision.
- A judge without a reported human-agreement rate is an instrument without a calibration.
- The gate must assert per-case regressions, not just an average — averages hide the expensive failures.
- Sample production continuously; alert separately on distribution drift and score drift.
- Report one agreed business metric on a cadence, traceable to the runs behind it.