A customer says the system “should be accurate”. That sentence is not a specification — it is the beginning of a negotiation, and how you run it decides whether the deployment is provable or arguable. Four acts, on one realistic case: pick the metrics, discover the real accuracy ceiling by measuring how often the experts agree with each other, calibrate an LLM judge against human grades, then set a CI gate and watch it catch a change that improves the average while quietly breaking the expensive cases. The method from Evals for Client Work, made playable.
metrics · inter-expert agreement · judge calibration · the gate that catches what an average hides
golden set
Real, stratified, frozen cases with agreed-correct answers. Changing it is a decision, not a fix.
inter-expert agreement
How often two humans doing the job give the same answer. Your practical ceiling — and nobody measures it.
judge calibration
How often the LLM judge agrees with a human on the same case. Report it beside every score or the score means nothing.
the gate
Thresholds in CI. Until an eval blocks a merge it is a document, not a control.
invoice-routing — eval.build
Act 1 of 4
The customer routes incoming invoices to one of six queues. Two senior clerks do it by hand today. They want it automated, and they want it “accurate”.
1/4
act
0
score
—
ceiling
Why the ceiling is the whole game
Every unhappy AI deployment contains the same conversation, six months in: the customer says it is not accurate enough, you show them a number, and nobody can agree whether the number is good. The way out is to establish the ceiling before anything is promised.
01
Two experts, forty cases, one afternoon
Have them label independently, then measure how often they match. This is the cheapest, most under-used measurement in enterprise AI, and it reframes every subsequent conversation about quality.
02
The disagreements are the specification
Put the experts in a room with the cases they split on. The argument they have is the rule nobody had written down. That resolution becomes the labelling guide, the judge prompt, and the acceptance criteria in the contract.
03
Targets are set against humans, not against 100%
“Match or exceed the 72% expert-agreement baseline, with under 2% in the high-cost error class” is testable, defensible and achievable. “95% accurate” is a number from a sales meeting that will be used against you.
04
Not all errors cost the same
Routing a routine invoice to the wrong queue costs minutes. Auto-approving one that needed legal review costs a great deal more. A single accuracy figure over a domain with asymmetric costs is actively misleading, and sophisticated buyers know it.
The gate, in code
Act 4 is a simulation of this. The third assertion is the one that earns its keep — an overall average can improve while the rare, expensive cases silently break.
# eval_gate.py — runs on every PR, including every prompt change
res = run_golden_set(system, cases)
assert res.overall >= 0.72, "below the agreed human baseline"assert res.high_cost_error_rate <= 0.02, "error budget blown"assert res.regressions == [], "cases that used to pass now fail"assert res.p95_latency_ms <= 4000 and res.cost_per_case <= 0.04
golden set
100–300
target from
human baseline
report beside score
judge agreement
gate that saves you
per-case
Building an eval, step by step
The two numbers that decide whether a golden set can be trusted: how well the human labelers agree beyond chance, and how wide the error bars on your score are. Both are small calculations worth doing before the first prompt is tuned.
01
Collect and label
Sample real cases, including the awkward ones, and have two experts label them independently. Keep their labels, not just the merged answer.
02
Measure agreement beyond chance
Raw agreement overstates it, since two labelers who both say "approve" most of the time agree often by luck. Cohen's kappa is (po − pe) / (1 − pe), where po is observed agreement and pe the agreement expected if each labeler answered independently at their own rates.
03
Set the target from the ceiling
Resolve the disagreements, publish the labelling guide, then set the target against the human baseline, not against 100%.
04
Size the set for the claim
A score on n cases has a margin of about 1.96 · √(p(1 − p) / n). Pick n so the margin is smaller than the differences you need to detect.
Worked example, 40 invoices. Two clerks label each as approve or review. Both approve 18, A approves and B reviews 4, A reviews and B approves 3, both review 15.
quantity
value
how
observed agreement po
0.825
(18 + 15) / 40
A approves, B approves
0.55, 0.525
22/40, 21/40
chance agreement pe
0.5025
0.55 × 0.525 + 0.45 × 0.475
Cohen's kappa
0.648
(0.825 − 0.5025) / (1 − 0.5025)
82.5% raw agreement is only 0.65 beyond chance, which is substantial but not clean. Treat a kappa near 0.4 or below as a sign the task is underspecified, and fix the guide before measuring any model.
cases n
score
95% interval
40
72%
57% to 84%
100
72%
63% to 80%
400
72%
67% to 76%
1,000
72%
69% to 75%
These are Wilson intervals around a 72% score. On 100 cases a system at 72% cannot be told apart from one at 78%. Holding the margin to 3 points takes 861 cases. For the rare, expensive error class the arithmetic is harsher: zero failures in 100 cases is still consistent with a true rate up to about 3% (the rule of three, 3/n), so verifying a 2% budget needs more than 150 cases. Stratify the golden set so each high-cost class has its own quota, and re-label a sample each quarter to catch drift.
Common questions
What accuracy should I promise a customer?
Whatever the measured inter-expert agreement supports, expressed as a target relative to it, with a separate cap on high-cost errors. Promising a number above the rate at which two humans agree with each other guarantees a fight you will lose later.
How big does a golden set need to be?
Usually 100–300 cases, stratified so the rare high-cost cases are over-represented. Every case should have been looked at by someone who knew what they were doing — that matters far more than volume.
Can I trust an LLM judge?
Only with a calibration number attached. Grade the same cases with humans and with the judge, report how often they agree, and inspect every disagreement. Give the judge the rubric verbatim, keep it blind to which system produced the output, use a small discrete scale, and pin the model version — a judge upgrade is a measurement change.
Where do the labels come from?
Best: work the customer’s team already did, with outcomes attached. Next best: a structured labelling session. Worst: you, guessing, the night before a demo. Better still, build a one-click “this was wrong” control into the product and let six weeks of real usage generate them.