FDE · Evals

Build the Gate That Decides “Done”.

A customer says the system “should be accurate”. That sentence is not a specification — it is the beginning of a negotiation, and how you run it decides whether the deployment is provable or arguable. Four acts, on one realistic case: pick the metrics, discover the real accuracy ceiling by measuring how often the experts agree with each other, calibrate an LLM judge against human grades, then set a CI gate and watch it catch a change that improves the average while quietly breaking the expensive cases. The method from Evals for Client Work, made playable.

metrics · inter-expert agreement · judge calibration · the gate that catches what an average hides
golden set

Real, stratified, frozen cases with agreed-correct answers. Changing it is a decision, not a fix.

inter-expert agreement

How often two humans doing the job give the same answer. Your practical ceiling — and nobody measures it.

judge calibration

How often the LLM judge agrees with a human on the same case. Report it beside every score or the score means nothing.

the gate

Thresholds in CI. Until an eval blocks a merge it is a document, not a control.

invoice-routing — eval.build
Act 1 of 4
The customer routes incoming invoices to one of six queues. Two senior clerks do it by hand today. They want it automated, and they want it “accurate”.
1/4
act
0
score
ceiling

Why the ceiling is the whole game

Every unhappy AI deployment contains the same conversation, six months in: the customer says it is not accurate enough, you show them a number, and nobody can agree whether the number is good. The way out is to establish the ceiling before anything is promised.

01

Two experts, forty cases, one afternoon

Have them label independently, then measure how often they match. This is the cheapest, most under-used measurement in enterprise AI, and it reframes every subsequent conversation about quality.

02

The disagreements are the specification

Put the experts in a room with the cases they split on. The argument they have is the rule nobody had written down. That resolution becomes the labelling guide, the judge prompt, and the acceptance criteria in the contract.

03

Targets are set against humans, not against 100%

“Match or exceed the 72% expert-agreement baseline, with under 2% in the high-cost error class” is testable, defensible and achievable. “95% accurate” is a number from a sales meeting that will be used against you.

04

Not all errors cost the same

Routing a routine invoice to the wrong queue costs minutes. Auto-approving one that needed legal review costs a great deal more. A single accuracy figure over a domain with asymmetric costs is actively misleading, and sophisticated buyers know it.

The gate, in code

Act 4 is a simulation of this. The third assertion is the one that earns its keep — an overall average can improve while the rare, expensive cases silently break.

# eval_gate.py — runs on every PR, including every prompt change res = run_golden_set(system, cases) assert res.overall >= 0.72, "below the agreed human baseline" assert res.high_cost_error_rate <= 0.02, "error budget blown" assert res.regressions == [], "cases that used to pass now fail" assert res.p95_latency_ms <= 4000 and res.cost_per_case <= 0.04
golden set
100–300
target from
human baseline
report beside score
judge agreement
gate that saves you
per-case

Common questions

What accuracy should I promise a customer?

Whatever the measured inter-expert agreement supports, expressed as a target relative to it, with a separate cap on high-cost errors. Promising a number above the rate at which two humans agree with each other guarantees a fight you will lose later.

How big does a golden set need to be?

Usually 100–300 cases, stratified so the rare high-cost cases are over-represented. Every case should have been looked at by someone who knew what they were doing — that matters far more than volume.

Can I trust an LLM judge?

Only with a calibration number attached. Grade the same cases with humans and with the judge, report how often they agree, and inspect every disagreement. Give the judge the rubric verbatim, keep it blind to which system produced the output, use a small discrete scale, and pin the model version — a judge upgrade is a measurement change.

Where do the labels come from?

Best: work the customer’s team already did, with outcomes attached. Next best: a structured labelling session. Worst: you, guessing, the night before a demo. Better still, build a one-click “this was wrong” control into the product and let six weeks of real usage generate them.

Keep going

Finished this one? 0 / 55 Labs done

Explore the topic

See this alongside everything else on the same subject — handbooks, system designs, challenges and tools, in one place.

More Labs