Design an Eval Pipeline

Where an interviewer pokes next

Getting the boxes right is the easy half. These are the questions that separate a candidate who drew the diagram from one who has run the thing. Answer each one out loud before you open it.

Show all answers

1. 01

   Why measure inter-expert agreement before promising an accuracy target?

   Show the answer

   Because it establishes the practical ceiling, and almost nobody measures it. If two of the customer’s own senior people, labelling the same forty cases independently, agree only 72% of the time, then a promise of 95% is a promise to exceed human performance on a task where the humans themselves do not have a consistent answer — and you will be held to it. Measuring the ceiling costs one afternoon and changes every subsequent conversation: targets become defensible ("match or exceed the 72% baseline, with under 2% in the high-cost class") rather than aspirational. The disagreements are worth as much as the number: the cases the experts split on are the business rules nobody had written down, and resolving them produces the labelling guide, the judge rubric and the acceptance criteria simultaneously.

2. 02

   Why must the golden set be frozen and stratified?

   Show the answer

   Frozen, because a set that changes between runs makes every comparison meaningless — the score moves for reasons nobody can attribute, the trend line becomes fiction, and per-case regression tracking stops working entirely. The failure mode is rarely deliberate; it is a case being quietly edited because it was failing, on the reasonable-sounding grounds that it was wrong anyway. Treat any change as a reviewed decision with a recorded reason, exactly like a schema migration. Stratified, because uniform sampling reproduces the production distribution, in which the rare high-cost cases are rare — so the score is dominated by the easy majority and stays green while exactly the cases that cost the customer money break. Over-represent the expensive tail deliberately, and report it as its own number.

3. 03

   What does judge calibration actually mean, and why report it?

   Show the answer

   Calibration is the measured rate at which the LLM judge agrees with human graders on the same cases. Without it you have a number with unknown accuracy: you cannot distinguish a real regression from a judge quirk, and the first time a customer disputes a score you have no answer. So grade a subset with humans, publish the agreement rate beside every score the judge produces, and inspect every disagreement — the disagreements usually reveal an ambiguous rubric rather than a bad judge. Practical rules follow from treating the judge as an instrument: give it the rubric verbatim, keep it blind to which system produced the output, use a small discrete scale rather than a continuous one, and pin the model version, because upgrading the judge is a measurement change that re-baselines every historical score.

4. 04

   Why is per-case regression tracking the assertion that matters?

   Show the answer

   Because an aggregate can improve while specific cases silently break, and in a domain with asymmetric error costs those specific cases are usually the expensive ones. A prompt change might raise the overall score by two points by handling common cases slightly better, while breaking the handful of rare cases that carry real financial or regulatory consequence — and the average absorbs it invisibly. Keeping per-case history in a results store makes "which cases used to pass and now fail?" answerable, and asserting that list is empty catches the class of failure no aggregate metric can. It is also the assertion that makes engineers trust the gate, because when it fires it points at something concrete rather than at a number that moved.

5. 05

   What breaks first after launch, and how do you detect it?

   Show the answer

   The assumption that the test set still resembles reality. The customer’s document mix shifts, a new vendor sends a different format, users start asking questions the system was never evaluated on, and the model provider ships an update — all while the gate stays green. Detection means sampling production continuously and scoring it, with two separate alerts: input-distribution drift (the cases changed) and score drift (the answers got worse). They have different causes and different fixes, and collapsing them into one alert makes both harder to act on. New failure classes found this way get promoted into the golden set as a reviewed decision, which is how the set stays honest without becoming a moving target. In a disconnected or air-gapped deployment none of this telemetry can leave, so the equivalent design is a golden set that lives inside the boundary and is run by the customer’s own operators, with only aggregate scores coming back out.