A customer says the system “should be accurate”. That sentence is not a specification — it is the beginning of a negotiation, and how you run it decides whether the deployment is provable or arguable. Four acts, on one realistic case: pick the metrics, discover the real accuracy ceiling by measuring how often the experts agree with each other, calibrate an LLM judge against human grades, then set a CI gate and watch it catch a change that improves the average while quietly breaking the expensive cases. The method from Evals for Client Work, made playable.
metrics · inter-expert agreement · judge calibration · the gate that catches what an average hides
golden set
Real, stratified, frozen cases with agreed-correct answers. Changing it is a decision, not a fix.
inter-expert agreement
How often two humans doing the job give the same answer. Your practical ceiling — and nobody measures it.
judge calibration
How often the LLM judge agrees with a human on the same case. Report it beside every score or the score means nothing.
the gate
Thresholds in CI. Until an eval blocks a merge it is a document, not a control.
invoice-routing — eval.build
Act 1 of 4
The customer routes incoming invoices to one of six queues. Two senior clerks do it by hand today. They want it automated, and they want it “accurate”.
1/4
act
0
score
—
ceiling
Why the ceiling is the whole game
Every unhappy AI deployment contains the same conversation, six months in: the customer says it is not accurate enough, you show them a number, and nobody can agree whether the number is good. The way out is to establish the ceiling before anything is promised.
01
Two experts, forty cases, one afternoon
Have them label independently, then measure how often they match. This is the cheapest, most under-used measurement in enterprise AI, and it reframes every subsequent conversation about quality.
02
The disagreements are the specification
Put the experts in a room with the cases they split on. The argument they have is the rule nobody had written down. That resolution becomes the labelling guide, the judge prompt, and the acceptance criteria in the contract.
03
Targets are set against humans, not against 100%
“Match or exceed the 70% expert-agreement baseline, with under 2% in the high-cost error class” is testable, defensible and achievable. “95% accurate” is a number from a sales meeting that will be used against you.
04
Not all errors cost the same
Routing a routine invoice to the wrong queue costs minutes. Auto-approving one that needed legal review costs a great deal more. A single accuracy figure over a domain with asymmetric costs is actively misleading, and sophisticated buyers know it.
The gate, in code
Act 4 is a simulation of this. The third assertion is the one that earns its keep — an overall average can improve while the rare, expensive cases silently break.
# eval_gate.py — runs on every PR, including every prompt change
res = run_golden_set(system, cases)
assert res.overall >= 0.70, "below the agreed human baseline"assert res.high_cost_error_rate <= 0.02, "error budget blown"assert res.regressions == [], "cases that used to pass now fail"assert res.p95_latency_ms <= 4000 and res.cost_per_case <= 0.04
golden set
100–300
target from
human baseline
report beside score
judge agreement
gate that saves you
per-case
Building an eval, step by step
The two numbers that decide whether a golden set can be trusted: how well the human labelers agree beyond chance, and how wide the error bars on your score are. Both are small calculations worth doing before the first prompt is tuned.
01
Collect and label
Sample real cases, including the awkward ones, and have two experts label them independently. Keep their labels, not just the merged answer.
02
Measure agreement beyond chance
Raw agreement overstates it, since two labelers who both say "approve" most of the time agree often by luck. Cohen's kappa is (po − pe) / (1 − pe), where po is observed agreement and pe the agreement expected if each labeler answered independently at their own rates.
03
Set the target from the ceiling
Resolve the disagreements, publish the labelling guide, then set the target against the human baseline, not against 100%.
04
Size the set for the claim
A score on n cases has a margin of about 1.96 · √(p(1 − p) / n). Pick n so the margin is smaller than the differences you need to detect.
Worked example, 40 invoices. Two clerks label each as approve or review. Both approve 18, A approves and B reviews 4, A reviews and B approves 3, both review 15.
quantity
value
how
observed agreement po
0.825
(18 + 15) / 40
A approves, B approves
0.55, 0.525
22/40, 21/40
chance agreement pe
0.5025
0.55 × 0.525 + 0.45 × 0.475
Cohen's kappa
0.648
(0.825 − 0.5025) / (1 − 0.5025)
82.5% raw agreement is only 0.65 beyond chance, which is substantial but not clean. Treat a kappa near 0.4 or below as a sign the task is underspecified, and fix the guide before measuring any model.
cases n
score
95% interval
40
72%
57% to 84%
100
72%
63% to 80%
400
72%
67% to 76%
1,000
72%
69% to 75%
These are Wilson intervals around a 72% score. On 100 cases a system at 72% cannot be told apart from one at 78%. Holding the margin to 3 points takes 861 cases. For the rare, expensive error class the arithmetic is harsher: zero failures in 100 cases is still consistent with a true rate up to about 3% (the rule of three, 3/n), so verifying a 2% budget needs more than 150 cases. Stratify the golden set so each high-cost class has its own quota, and re-label a sample each quarter to catch drift.
The picture: what the gate sees on one pull request
Act 4 of the lab, as the pipeline that runs on every pull request. The golden set and the change meet in one run, the calibrated judge grades what cannot be string-matched, and four independent assertions read the per-case results. Two of them fail on a change that improved the headline number. An overall-only gate would have shown a green tick here, which is the whole argument for per-case and per-class assertions. Latency passed too: a gate that checked only speed and the average would have shipped the change.
Common mistakes when building an eval
The page covers the big three: promising more than the experts agree on, trusting an uncalibrated judge, and gating on the average. These are the quieter ones. Each passes review because it looks like diligence.
Tuning prompts against the gate set
Every prompt change you keep because it raised the golden-set score fits the set a little more closely. After forty rounds the score measures the set, not the system. Keep a development slice for iteration and touch the frozen gate set only to decide. On 100 cases a 72% score already carries a 63% to 80% interval, so a few points of “improvement” found by search is mostly noise.
Comparing two averages instead of the flips
Overall went from 74% to 77%. On 100 cases that might be 7 cases flipping from fail to pass and 4 from pass to fail. A paired test such as McNemar’s looks only at those 11: (|7 − 4| − 1)² / 11 = 0.36, far below the 3.84 needed at the 5% level. The averages hide that the change was close to a coin toss, and they hide which 4 cases broke.
Reporting a stratified score as if it were production
Over-sampling legal invoices makes their errors visible, but it skews the overall figure. Say the set is 150 routine and 50 legal, the system scores 80% and 60%, and production is 95% routine (illustrative). The set reports 75%; production would see 79%. Report per class, and re-weight to the production mix whenever you quote one overall number.
Labelling with the model’s answer on screen
Reviewers shown a suggested answer tend to accept it, so labels drift toward whatever the current system says and the eval grades the system against itself. Collect gold labels blind, before anyone sees a model output. If you must use model pre-labels to save time, measure how often reviewers change them, and audit a blind sample. The lab’s Act 2 has the clerks label independently for the same reason: a second label that has seen the first is not a second opinion.
Changing the set and the system in one pull request
A PR that edits the prompt and also “fixes” three golden labels produces a score change nobody can attribute. Version the golden set, the rubric and the judge model like code, change one at a time, and re-run the old system on a new set version so the baseline moves with it. And capture the human baseline, time per invoice and error rate, in week one: once the system ships, the old workflow is gone and so is the number you would compare against.
Check yourself
Four questions on the lab’s own numbers. Work each one out before opening it; the arithmetic is the point.
The two clerks agree on 7 of 10 invoices. Can you write “match the 70% expert baseline” into the contract today?
Not yet. The 95% interval around 7 of 10 runs from about 40% to 89%, so 70% is a guess. Ten cases are enough to find the disagreements worth discussing, not to fix a baseline. Label the full 40 to 100 cases first, then write the target against what that measures.
Two labelers both say “approve” 95% of the time and agree on 92% of cases. Is that a reliable labelling task?
No. Chance agreement is 0.95² + 0.05² = 0.905, so kappa = (0.92 − 0.905) / (1 − 0.905) ≈ 0.16, barely above luck. Almost all the 92% comes from both saying “approve” by default. The rare “review” cases, the ones that matter, are where they disagree. Fix the guide for those before measuring a model.
A change lifts overall accuracy from 74% to 77% and raises the high-cost error rate from 1.1% to 2.6%. The author proposes loosening the 2% budget to 3% “since accuracy went up”. What do you do?
Keep the budget and block the merge. The budget exists because those errors cost far more than the routine ones the change fixed, so trading them is exactly the trade the contract forbids. Read the regression list (inv-103, inv-118, inv-140), find what the prompt did to legal invoices, and fix that.
Your stratified set holds 300 high-cost cases and the system makes zero high-cost errors on them. Does that support a 2% budget?
Yes. By the rule of three, zero failures in n cases bounds the true rate at about 3/n with 95% confidence: 3/300 = 1%, under the 2% budget. With 100 cases the same zero would only bound it at 3%, which cannot confirm a 2% budget.
Common questions
What accuracy should I promise a customer?
Whatever the measured inter-expert agreement supports, expressed as a target relative to it, with a separate cap on high-cost errors. Promising a number above the rate at which two humans agree with each other guarantees a fight you will lose later.
How big does a golden set need to be?
Usually 100–300 cases, stratified so the rare high-cost cases are over-represented. Every case should have been looked at by someone who knew what they were doing — that matters far more than volume.
Can I trust an LLM judge?
Only with a calibration number attached. Grade the same cases with humans and with the judge, report how often they agree, and inspect every disagreement. Give the judge the rubric verbatim, keep it blind to which system produced the output, use a small discrete scale, and pin the model version — a judge upgrade is a measurement change.
Where do the labels come from?
Best: work the customer’s team already did, with outcomes attached. Next best: a structured labelling session. Worst: you, guessing, the night before a demo. Better still, build a one-click “this was wrong” control into the product and let six weeks of real usage generate them.