From “it should be
accurate” to a gate
Evals are the single most-cited hard skill across frontier-lab FDE job descriptions and the least taught anywhere. Not because the maths is hard — because the hard part is social: you have to get people who disagree with each other to write down what correct means, and then make that survive a procurement conversation. Here is the whole method, with code.
Why this is the FDE skill
An eval in client work is not a benchmark. It is the written, executable definition of done for a deployment — the thing that decides whether the customer pays, whether you can ship a change, and whether the renewal conversation is a negotiation or a formality.
Three facts explain why this lands on the FDE rather than on a data scientist. First, only the FDE is in the room with the people who actually know what correct looks like. Second, the eval is a commercial artefact as much as a technical one — it usually ends up referenced in the statement of work. Third, without one, every disagreement about quality becomes an argument about anecdotes, and the loudest anecdote wins.
The market context makes it sharper: MIT’s NANDA study found roughly 95% of enterprise AI pilots produced no measurable profit-and-loss impact.17 “No measurable” is doing a lot of work in that sentence. A large share of those pilots did not fail to work — they failed to demonstrate that they worked, because nobody defined the measurement before the money was spent.
“Before we build anything, can we spend two hours agreeing how we will know it works?” It sounds like process overhead. It is the highest-return two hours of the engagement, and it is the thing that separates a deployment that renews from one that becomes a case study in the 95%.
Step 1 — negotiate what “correct” means
This is the step nobody writes about and the one that decides everything downstream.
You will be told “it should be accurate”. Accuracy is not a property of a model; it is a property of an agreement. And in every enterprise domain worth deploying into, the domain experts do not agree with each other.
Measure inter-expert agreement first
Take 40 real cases. Have two or three experts label them independently. Then measure how often they agree with each other. This costs an afternoon and changes the entire engagement.
If two senior claims adjusters agree 72% of the time, no model is reaching 95%, and everyone in the room now knows it — before it is written into a contract.
Resolve the disagreements into a rubric
Sit the experts down with the cases they split on. The argument they have is the specification you were missing. Write the resolution as rules: “if the invoice has no PO number, route to manual review, do not guess.”
That rubric becomes three things at once: the labelling guide, the LLM-judge prompt, and the acceptance criteria language in the statement of work.
Set the target against the human baseline, not against 100%
“Match or exceed the expert-agreement rate of 72% on the held-out set, with under 2% of cases in the high-cost error class” is a target you can hit, defend and test. “95% accurate” is a number someone invented in a sales meeting.
Not all errors cost the same. Approving a fraudulent claim and rejecting a valid one are both “errors” and one of them is fifty times more expensive. Get the cost ratio in writing, then pick the operating point from it. An eval that reports a single accuracy number for a domain with asymmetric costs is actively misleading.
Step 2 — build a golden set that survives contact
A golden set is a fixed collection of inputs with agreed-correct outputs. Three properties make one useful, and the third is the one people skip.
| Property | What it means | How it goes wrong |
|---|---|---|
| Real | Drawn from the customer’s actual traffic, including the ugly cases. | Synthetic examples that share none of the real distribution’s weirdness. |
| Stratified | Deliberately over-samples the rare, high-cost cases. | A random sample where the 2% case that matters appears once. |
| Frozen and versioned | Checked in, with a changelog. Changing it is a decision, not a fix. | Quietly editing a label because the model got it wrong. This is how a gate becomes theatre. |
Sizing: 100–300 cases is usually enough to make a go/no-go decision, and far more useful than 10,000 mediocre ones. You want every case to have been looked at by a human who knew what they were doing.
{"id":"c-0142","input":{"doc":"invoice_8891.pdf"},"expect":{"route":"manual","reason":"no PO"},"tags":["rare","high-cost"]}
{"id":"c-0143","input":{"doc":"invoice_8892.pdf"},"expect":{"route":"auto"},"tags":["common"]}
Where the labels come from matters as much as the labels. The best source is work the customer’s team already did — historical decisions with outcomes attached. The second best is the labelling session from step 1. The worst is you, guessing, at midnight, because the demo is tomorrow.
Build a one-click “this was wrong” control into whatever you ship, and log the correction. Six weeks of real usage then produces a better golden set than any labelling exercise, and it costs the user two seconds. This is the highest-leverage feature most FDE deployments never build.
Step 3 — calibrate the judge before you trust it
An LLM judge is a measuring instrument. Instruments get calibrated against a reference before anyone reports numbers from them.
For fuzzy outputs — summaries, extracted fields, agent transcripts — exact matching does not work, so you grade with a model. The mistake is treating the judge’s output as ground truth on day one. It is a hypothesis about agreement with humans, and it has to be tested like one.
# 2. grade the SAME cases with the judge
# 3. compare judge vs human — this number is the thing you report
agree = sum(j == h for j, h in zip(judge_grades, human_grades)) / len(human_grades)
# and look at the disagreements individually. they are always informative:
# either the rubric is ambiguous, or the judge prompt is leading it.
Judge prompts that behave
- Give the rubric, verbatim, from step 1b — not a paraphrase.
- Ask for a label plus a reason; the reasons are how you debug disagreements.
- Use a small discrete scale (pass/fail, or 1–3). Continuous scores invent precision.
- Never show the judge which system produced the output.
- Pin the judge model and version, and note it in the report. A judge upgrade is a measurement change.
Failure modes to check for
- Position bias in pairwise comparisons — swap the order and re-run.
- Length bias — judges reward verbosity unless the rubric forbids it.
- Self-preference — a judge from the same family as the generator grades it kindly.
- Rubric drift — someone “improves” the prompt and every historical number becomes incomparable.
Report the calibration number alongside every result. “Judge agrees with our experts 89% of the time, and here is the model’s score under that judge” is a defensible sentence in front of a procurement team. A bare score is not.
Step 4 — wire the gate
An eval that is not in CI is a document. Once it blocks a merge, it is a control.
res = run_golden_set(system, cases)
assert res.overall >= 0.72, f"below the agreed human baseline: {res.overall:.2f}"
assert res.high_cost_error_rate <= 0.02, "high-cost error budget blown"
assert res.regressions == [], f"cases that used to pass now fail: {res.regressions}"
# the third assertion is the one that saves you. an overall average can
# stay flat while the rare, expensive cases quietly break.
| Gate | Threshold from | Blocks what |
|---|---|---|
| Overall score | The human agreement baseline | A change that makes the system worse on average. |
| High-cost error budget | The cost asymmetry from step 1 | A change that trades many cheap wins for one expensive loss. |
| Per-case regression list | The previous run | Silent breakage of specific cases the customer already saw working. |
| Cost and latency | The deployment’s budget | A quality win that quadruples the bill or breaks the SLA. |
Prompts are code. A prompt edit that skips the eval gate is an unreviewed production deploy, and it is the single most common source of “it got worse and nobody knows when” in deployed LLM systems. Put prompts in version control and run the gate on every change to them.
Build one end to end
Take a fuzzy goal, choose metrics, label a golden set, calibrate a judge against human grades and set the CI gate — interactively, in about fifteen minutes.
Step 5 — evals for agents are different
Agent deliverables are now standard in FDE work — frontier-lab job descriptions list MCP servers, sub-agents and agent skills as things you hand to a customer.10 Grading a single output does not describe an agent’s behaviour, because an agent can reach the right answer through an unacceptable path.
What to measure
- Task success — did the end state match the intent?
- Trajectory validity — were the tool calls sensible and permitted?
- Cost and steps — right answer in 40 tool calls is a failure.
- Escalation correctness — did it hand off when it should have?
- Damage — did it write anything it should not have?
How to run it
- Record real trajectories and replay them against changes.
- Sandbox the tools so an eval run cannot touch production.
- Seed everything you can; report variance across repeats rather than a single run.
- Keep a hostile set: prompt injection, contradictory instructions, missing data.
Not accuracy — escalation quality. An agent that handles 60% of cases and cleanly hands off the rest is deployable. One that handles 85% and confidently mishandles the remainder is not, and the difference will surface in front of their customers rather than yours.
Step 6 — hand the eval over
The eval outlives the engagement. Whether it survives your departure decides whether the deployment does.
Drift monitoring, not just a gate
The gate catches your changes. Drift catches theirs: an upstream schema change, a new document type, a seasonal shift in the input mix. Track input distribution and output distribution over time and alert on movement, not just on failures. Most post-handoff degradation is invisible to a CI gate because nothing in the repo changed.
A runbook their team can execute
How to add a case to the golden set, how to re-run the suite, what each threshold means, who decides when a threshold moves, and what to do when the gate fails. Two pages. Written for someone who was not in any of your meetings.
The business review runs on this
At renewal, the eval results plus adoption telemetry are the argument. That is the mechanism connecting this handbook to the commercial outcome — see Proving Value & ROI.
Their engineer can add a case, re-run the suite and read the result without calling you. If that is not true, you did not hand over an eval — you handed over a dependency on yourself, which is the definition of a failed handoff.
Quick answers
What is an eval in an AI deployment?
A written, executable definition of done: a frozen set of real cases with agreed-correct outputs, a way to grade the system against them, and thresholds that decide whether a change can ship. In client work it is a commercial artefact as much as a technical one, because it usually ends up referenced in the statement of work and in the renewal conversation.
How do you decide what accuracy target to promise a customer?
Measure inter-expert agreement first. Have two or three domain experts independently label about 40 real cases and measure how often they agree with each other. That agreement rate is the practical ceiling, and a target expressed against it — match or exceed the human baseline, with a separate cap on high-cost errors — is both achievable and defensible. A promised 95% invented in a sales meeting is neither.
How do you calibrate an LLM judge?
Grade the golden set with humans, grade the same cases with the judge, then report how often the judge agrees with the humans. Inspect every disagreement individually — each one means either the rubric is ambiguous or the judge prompt is leading. Report the calibration number alongside every result, and pin the judge model and version, because a judge upgrade is a measurement change.
What should an eval gate block in CI?
Four things: an overall score below the agreed human baseline, a high-cost error rate above the agreed budget, any per-case regression where something that used to pass now fails, and a cost or latency breach. The per-case regression check matters most, because an overall average can stay flat while the rare expensive cases quietly break.
How are evals for agents different?
A single output grade does not describe an agent, because it can reach the right answer through an unacceptable path. Measure task success, trajectory validity, cost and step count, escalation correctness and whether it wrote anything it should not have. Record real trajectories and replay them, sandbox the tools, and keep a hostile set covering prompt injection, contradictory instructions and missing data.
Sources
Every number on this page traces to one of these. Where a figure is self-reported or crowd-sourced rather than first-party, it is labelled inline.
Cited on this page
- OpenAI — Forward Deployed Engineer job description — “full arc of a deployment”. openai.com/careers/forward-deployed-engineer-(fde)-nyc-new-york-city/
- Anthropic — Forward Deployed Engineer, Applied AI (JD) — 40/30/30 split; MCP servers, sub-agents and agent skills as deliverables. jobs.menlovc.com/companies/anthropic/jobs/69674588-forward-deployed-engineer-applied-ai
- MIT NANDA “State of AI in Business” — 95% of pilots with no measurable P&L impact — widely reported; figure cited via press coverage rather than a public PDF. techcrunch.com/2026/07/30/forward-deployed-engineers-are-the-ai-industrys-latest-talent-obsession/
- Distyl AI — “Interviewing with AI at Distyl” (first-party) — AI-assisted take-home and the anti-patterns they reject. distyl.ai/blog/frontier/interviewing-with-ai-at-distyl
Keep going
Explore the topic
See this alongside everything else on the same subject — handbooks, system designs, challenges and tools, in one place.
More Handbooks
- The Prompting HandbookA friendly, hands-on field guide for everyday humans — learn the CRISP framework, spot bad prompts, practice with real recipes, play a drag-and-drop game, and test yourself with a quiz. No code required.Read →
- The Agentic AI Interview HandbookTwenty topics every senior AI engineer should be able to reason about live — from eval pipelines to reliability patterns for generative systems.Read →
- The Senior AI Engineer Interview Handbook60 questions across architecture, production incidents, agentic systems, RAG, evals, cost, safety, and leadership — what staff-level AI interviewers actually probe for.Read →
- 50 Angular Interview QuestionsA visual handbook covering components, change detection, RxJS, signals, routing, forms, performance, and testing — what interviewers actually probe for in senior Angular roles.Read →
- 50 Python Interview QuestionsFundamentals to advanced: data structures, OOP, iterators & generators, the GIL, asyncio, memory, testing, and the standard library — a visual walk through everything a Python interview touches.Read →
- 51 LLM Evals Interview QuestionsGolden sets, LLM-as-judge, regression testing, offline vs online evals, RAG evals, agent evals, red-teaming, and observability — demystified for interviews and production.Read →
Explore more from Vibe Engines
Get the next one in your inbox.
New handbooks, system-design walkthroughs, and tools — straight to your inbox. No spam, unsubscribe anytime.