Evals & Metrics

Every metric you need to evaluate LLMs, RAG and agents — what it measures and when it lies.

An eval is a test set, a metric and a threshold that together decide whether a change ships. Metrics are proxies: every one on this sheet can be gamed or can drift from what users actually experience, so pick the cheapest metric that tracks the outcome you care about, and check it against human judgment.

Measure each stage separately — retrieval, generation, the agent’s actions — so a failure points at the part to fix. And report uncertainty: on a few hundred examples, a two-point gain is usually noise.

Classification & generation

Accuracy / F1: exact tasks only; F1 = harmonic mean of precision & recall
Token F1: partial-credit overlap for QA answers — softer than exact match
Perplexity: how surprised the model is by text; compares LMs, not products
pass@k: code: any of k samples passes the tests — the coding-eval standard
BLEU / ROUGE: n-gram overlap (translation/summaries) — legacy; weak on meaning
Exact match: normalized string equality — strict and cheap, right for short factual answers
BERTScore: similarity of contextual embeddings — credits paraphrases that n-gram metrics miss
chrF: character n-gram F-score — sturdier than BLEU for morphologically rich languages
WER / CER: speech and OCR: word or character edits divided by reference length; lower is better

Precision, recall & friends

Precision: TP / (TP + FP): of everything flagged, how much was right
Recall: TP / (TP + FN): of everything that was there, how much was found
Confusion matrix: TP, FP, FN, TN counts — every classification metric is a ratio of these four
Macro vs micro average: macro averages per-class scores, so rare classes count; micro pools every decision
ROC-AUC vs PR-AUC: with rare positives, PR-AUC shows the weakness that ROC-AUC hides
Calibration (ECE): do 80%-confident predictions come true 80% of the time? ECE measures the gap
Cohen’s kappa: annotator agreement beyond chance; 0.61–0.80 is conventionally “substantial”

RAG & retrieval

Recall@k: did the gold chunk make the top k — the retriever’s core number
MRR / nDCG: rank-position quality: reciprocal rank / graded, discounted gain
Faithfulness: is every answer claim supported by the retrieved context (judge-scored)
Answer relevance: does the answer address the question (independent of truth!)
Context precision: how much of what you stuffed into the prompt was actually needed
Precision@k: how many of the top k results are relevant — the noise handed to the model
Context recall: are all the facts the reference answer needs present in the retrieved context?
Citation accuracy: does each cited passage actually support the sentence that cites it?
Split the pipeline: score retrieval and generation separately, so a failure points at the right stage

LLM-as-judge

Rubric scoring: 1-5 against explicit written criteria; anchor each grade with examples
Pairwise: A vs B beats absolute scores — judges are better at comparing
Known biases: position (swap and re-ask), verbosity, self-preference — control all three
Calibrate: judge vs human labels on 50-100 items BEFORE trusting it at scale
Reference-guided: give the judge a gold answer — far more reliable than asking “is this good?”
Binary checks: several yes/no criteria beat one 1–10 score: easier to agree on and to debug
Reasoning before score: have the judge explain first, and store the rationale to audit disagreements

Agents & production

Task success rate: did the whole episode achieve the goal — the only end-truth
Steps / cost per task: efficiency; watch for loops and tool-call thrash
SWE-bench style: real environment + held-out tests — evaluate in the world, not on text
Regression gate: golden set + threshold in CI; prompt changes ship only if evals hold
Online signals: thumbs, edits, retries, escalations — the eval set your users write
pass^k: reliability: all k independent trials succeed — much lower than pass@k for agents
Tool-call accuracy: right tool, right arguments, right order — catches lucky end-to-end passes
TTFT and p95 latency: time to first token plus tail latency — averages hide the slow requests
Cost per successful task: total tokens and tool calls divided by successes, not by attempts
Over-refusal rate: share of safe requests the model declines — the other side of safety evals

Statistics & hygiene

Confidence interval: 95% ≈ p ± 1.96·√(p(1−p)/n): n = 100 at 80% is about ±8 points
Sample size: a 2-point difference needs thousands of items; 50 only catch big regressions
Paired comparison: score both versions on the same items, compare per item — far tighter than two totals
Bootstrap: resample items with replacement to get an interval for any metric, no formula needed
Contamination: public benchmarks leak into training data — keep a private, recent test set
Overfitting the eval: tuning prompts on the test set until it passes — hold out a split you never inspect
Slice the results: report by category, language and length — an average hides a broken segment

Recipes

Precision, recall and F1 from a confusion matrix

spam filter on 1,000 emails:   TP = 80   FP = 20   FN = 40   TN = 860

precision = TP / (TP + FP) = 80 / 100 = 0.80
recall    = TP / (TP + FN) = 80 / 120 = 0.667
F1        = 2 * P * R / (P + R) = 2 * 0.80 * 0.667 / 1.467 = 0.727
accuracy  = (TP + TN) / total = 940 / 1000 = 0.94   <- looks great, yet a third of the spam gets through

Accuracy is dominated by the 860 easy negatives, which is why imbalanced problems report precision, recall or F1 instead. Decide which error costs more before choosing the threshold.

MRR and nDCG@3 for a retriever

query 1: first relevant result at rank 1   ->  1/1 = 1.00
query 2: first relevant result at rank 3   ->  1/3 = 0.33
query 3: nothing relevant in the top k     ->  0
MRR = (1.00 + 0.33 + 0) / 3 = 0.44

nDCG@3 with graded relevance of the top 3 = [3, 0, 2]
DCG    = 3/log2(2) + 0/log2(3) + 2/log2(4) = 3 + 0 + 1    = 4.00
IDCG   = ideal order [3, 2, 0] = 3/1 + 2/1.585 + 0          = 4.26
nDCG@3 = 4.00 / 4.26 = 0.94

MRR only cares about the first hit, which fits “find me the one document”. nDCG rewards putting the most relevant items highest; some libraries use the gain 2^rel − 1 instead of rel, so compare numbers from the same implementation.

The unbiased pass@k estimator

n = 10 samples per problem, c = 3 of them pass the hidden tests
pass@k = 1 - C(n - c, k) / C(n, k)

pass@1 = 1 - C(7, 1) / C(10, 1) = 1 - 7/10   = 0.30
pass@5 = 1 - C(7, 5) / C(10, 5) = 1 - 21/252 = 0.917
average this over every problem in the benchmark

Sampling n ≥ k and using the formula gives a lower-variance estimate than literally drawing k samples. The temperature used for sampling changes pass@k, so report it.

How big is the error bar on my eval?

n = 200 test items, 160 correct  ->  p = 0.80
standard error = sqrt(p * (1 - p) / n) = sqrt(0.8 * 0.2 / 200) = 0.028
95% interval   = 0.80 ± 1.96 * 0.028   = 0.80 ± 0.055   ->  74.5% to 85.5%

a new prompt scoring 82% on these 200 items is not clearly better:
compare the two per item (paired), or add items, before concluding

The formula assumes independent items. Near-duplicate items make the real interval wider than it looks, so deduplicate the test set first.

Calibrate an LLM judge before trusting it

1. hand-label 100 outputs pass/fail (two people; measure their agreement first)
2. run the judge prompt on the same 100 outputs
3. agreement = matching labels / 100; count false passes separately
4. read every disagreement, fix the rubric, re-run; keep 20 items aside to confirm
5. ship the judge only if it agrees with humans about as well as they agree with each other

False passes are the dangerous direction, because they let regressions ship. Re-run the calibration whenever the judge model or the rubric changes.

Questions about Evals & Metrics Cheat Sheet

What is the difference between precision and recall?

Precision asks how many of the items you flagged were actually correct: true positives divided by everything you predicted positive. Recall asks how many of the real positives you found: true positives divided by everything that was actually positive. A spam filter that flags only one email it is completely sure about has perfect precision and terrible recall, while one that flags everything has perfect recall and poor precision. Raising the decision threshold usually trades recall for precision. F1, the harmonic mean of the two, summarizes both when false positives and false negatives matter equally.

What is pass@k?

pass@k is the standard metric for code generation: the probability that at least one of k generated solutions passes a problem’s unit tests, averaged over the benchmark. It was popularized by the HumanEval paper, which also introduced an unbiased estimator: generate n samples, count the c that pass, and compute one minus the ratio of combinations C(n minus c, k) over C(n, k). pass@1 approximates the chance a single attempt works, while larger k reflects settings where a user or a test suite can pick the best of several candidates. For agents, the stricter pass^k requires all k trials to succeed.

How do you evaluate a RAG system?

Evaluate retrieval and generation separately, then end to end. For retrieval, build a set of questions with known relevant documents and measure recall@k, precision@k and MRR or nDCG, which tells you whether the right context reaches the model. For generation, measure faithfulness, meaning every claim is supported by the retrieved context, and answer relevance, usually with a calibrated LLM judge. End to end, check answer correctness against reference answers and track citation accuracy. Slice results by question type, because a failing category often hides inside a good average, and rerun the suite on every change.

Can I trust an LLM as a judge?

Only after you calibrate it. LLM judges are cheap and scale well, but they have known biases: they favor the first option in pairwise comparisons, prefer longer answers, and rate their own model family higher. Counter these by swapping the order and averaging, using explicit rubrics or reference answers, asking for several yes or no checks instead of one score, and using a different model as judge where possible. Then compare the judge with human labels on fifty to a hundred items, and trust it at scale only when its agreement with humans is close to the agreement between humans.

Go deeper

Agent Evals handbook →LLM Evals interview handbook →LLM Judge Builder →Challenge: Token F1 →Design LLM Eval & Observability →