Evals & Metrics
Every metric you need to evaluate LLMs, RAG and agents — what it measures and when it lies.
Classification & generation
- Accuracy / F1
- exact tasks only; F1 = harmonic mean of precision & recall
- Token F1
- partial-credit overlap for QA answers — softer than exact match
- Perplexity
- how surprised the model is by text; compares LMs, not products
- pass@k
- code: any of k samples passes the tests — the coding-eval standard
- BLEU / ROUGE
- n-gram overlap (translation/summaries) — legacy; weak on meaning
RAG & retrieval
- Recall@k
- did the gold chunk make the top k — the retriever’s core number
- MRR / nDCG
- rank-position quality: reciprocal rank / graded, discounted gain
- Faithfulness
- is every answer claim supported by the retrieved context (judge-scored)
- Answer relevance
- does the answer address the question (independent of truth!)
- Context precision
- how much of what you stuffed into the prompt was actually needed
LLM-as-judge
- Rubric scoring
- 1-5 against explicit written criteria; anchor each grade with examples
- Pairwise
- A vs B beats absolute scores — judges are better at comparing
- Known biases
- position (swap and re-ask), verbosity, self-preference — control all three
- Calibrate
- judge vs human labels on 50-100 items BEFORE trusting it at scale
Agents & production
- Task success rate
- did the whole episode achieve the goal — the only end-truth
- Steps / cost per task
- efficiency; watch for loops and tool-call thrash
- SWE-bench style
- real environment + held-out tests — evaluate in the world, not on text
- Regression gate
- golden set + threshold in CI; prompt changes ship only if evals hold
- Online signals
- thumbs, edits, retries, escalations — the eval set your users write