Vibe Engines
YouTube
LEARNING PATH · AI Engineering

Master LLM Evaluation

For engineers who need to prove their AI actually works.

Advanced ~2h 45m7 lessons8 steps

Shipping an LLM feature is easy; knowing whether it is good is the hard part. Build the vocabulary of evals, design a judge rubric, implement a metric by hand, then evaluate agents and a real production system.

  • Speak precisely about offline vs online evals, judges and rubrics
  • Design an LLM-as-judge rubric that resists gaming
  • Implement eval metrics like token-level F1 from scratch
  • Evaluate agents and conversational systems, not just single calls
0 / 7 done · 0%
  1. HandbookNext up

    51 LLM Evals Interview Questions

    The vocabulary of evals — start here.

  2. ToolTool · optional

    LLM-as-Judge Rubric Builder

    Design an LLM-as-judge rubric interactively.

  3. Challenge

    Token-Level F1

    Implement a real metric by hand, with tests.

  4. Handbook

    The Agent Evaluations Handbook

    Evaluate agents — multi-step, tool-using.

  5. Handbook

    Self-Improving Agents

    Measure the verifier itself, and report a time horizon instead of a score.

  6. AI System Design

    Design an AI Agent System

    The agent system your evals must judge.

  7. AI System Design

    Design LLM Eval & Observability

    Build the eval + observability system around production.

  8. Handbook

    The Senior AI Engineer Interview Handbook

    Tie evals into senior-scope AI engineering.

How this path works

This path is for engineers who have shipped, or are about to ship, an LLM feature and need to know whether it works. It suits AI engineers, backend engineers who inherited a prompt, and anyone preparing for interview questions about evaluation. It is rated advanced because it assumes you have already built something with a model and seen it fail in a way a quick demo did not predict.

Evaluation is where many LLM projects stall. Outputs are free-form, so exact-match tests rarely work. A judge model can be biased or drift. A single run tells you little when outputs vary. And a metric that looks good on a small dataset can hide real regressions. Interviewers probe this area for the same reason: it shows whether you have operated an AI system rather than just called one.

The steps run from vocabulary to practice to systems. You start with the question bank to get the terms right, then design a judge rubric and implement a metric by hand so you have touched both ends of the spectrum. Next come agents, which are harder to grade, and the idea that the verifier itself needs measuring. The last steps place evaluation inside a full production system and a senior-level view of the whole job.

What each step gives you

  1. 51 LLM Evals Interview Questions

    Fifty-one questions on golden sets and contamination, why BLEU and exact match fail on generated text, factuality, the biases of LLM judges and how to validate one, CI regression gates, and offline versus online evaluation. Treat it as your vocabulary list: if you cannot answer a question aloud in two sentences, that is the section to reread before moving on.

  2. LLM-as-Judge Rubric Builder

    Define criteria and a scoring scale, and the tool generates a copy-ready judge prompt with guidance on position bias, verbosity bias and ties built in. Using it forces you to be specific about what good means. It prepares you for the task of writing a rubric that an LLM can apply consistently, and for explaining how you would check that it does.

  3. Token-Level F1

    You implement token-level F1, the overlap metric used for question answering: lowercase and split both strings, count shared tokens as a multiset, then combine precision and recall. Writing it by hand shows exactly what the metric rewards and what it misses. Expect to be asked why a correct answer phrased differently scores poorly, and what you would use instead.

  4. The Agent Evaluations Handbook

    Grading agents means grading multi-step, tool-using behavior. This handbook builds the frame of task, agent and grader, compares code-based, model-based and human graders, and shows how to write rubrics a judge will follow. It also covers the math of non-determinism, including passing at least once in k tries versus passing every time, and capability versus regression suites.

  5. Self-Improving Agents

    A long handbook on how agents improve using their own runs: repeated sampling, why picking the right answer is a separate problem from finding one, and how to build and break a verifier. For this path, focus on the measurement ideas: estimating pass at k without bias, checking the verifier itself, and reporting a time horizon rather than a single score.

  6. Design an AI Agent System

    The plan, act and observe loop, tool calling, safe execution and memory, built up piece by piece. You read it here because you cannot design an evaluation for a system you do not understand. After it, you can list the places an agent can go wrong, which become the categories your test tasks and graders need to cover.

  7. Design LLM Eval & Observability

    Designs the system around production: tracing as the foundation, versioned golden datasets, programmatic, LLM-judge and human scorers, offline evals as a CI gate, online evaluation and drift, and user-signal feedback. It ends with interview-style questions, such as what happens when a judge and a human disagree. It answers where evaluation lives in a running product.

  8. The Senior AI Engineer Interview Handbook

    Sixty senior-level questions, including how to build an eval framework from scratch, evaluate retrieval separately from answers, test an agent before launch, and detect drift in production. Use it as a final check: answer the evaluation questions aloud, then read the rest to see how measurement connects to architecture, incidents and cost.

A study plan

  1. Session 1

    Read the first half of 51 LLM Evals Interview Questions. Write one paragraph defining golden set, regression suite and judge drift in your own words, using a feature you know as the example.

  2. Session 2

    Finish the question bank, then build a judge prompt with the LLM-as-Judge Prompt Builder for a real task. Run it on ten outputs you have also graded yourself and note where it disagrees with you.

  3. Session 3

    Solve Token-Level F1 without hints. Then compute it on twenty real answer pairs and read the five lowest scores to find cases where the metric is wrong and a human would disagree.

  4. Session 4

    Read The Agent Evaluations Handbook and the relevant sections of the Self-Improving Agents Handbook. Write ten tasks and a grader for an agent you use, and run each task several times to see the variance.

  5. Session 5

    Read the AI Agent System and Eval and Observability designs, then draw your own pipeline from trace to CI gate. Finish with the evaluation questions in the senior engineer set, answered aloud.

Questions about this path

How do you evaluate an LLM application?

Start with a golden dataset of realistic inputs and the outcomes you consider acceptable, then score outputs with the cheapest reliable method: exact checks where an answer is verifiable, an LLM judge with a clear rubric where it is not, and human review to calibrate the judge. Run the suite before every release and track results over time, then add online monitoring for drift. This path walks through each piece in order.

What is LLM-as-a-judge and can I trust it?

LLM-as-a-judge means using a model to grade another model's output against a rubric. It scales far better than human review but has known biases, such as favoring longer answers or the first option in a pair. You can trust it to the degree you have validated it: compare its scores with human labels on a sample, tighten the rubric where they disagree, and recheck whenever the judge model changes.

Why isn't accuracy enough to evaluate generated text?

Generated answers can be correct in many different wordings, so exact match and overlap metrics such as BLEU or ROUGE punish valid responses and reward superficially similar wrong ones. A single accuracy number also hides which kinds of inputs fail. Better practice is to combine task-specific checks, a calibrated judge, and slices of your dataset by input type, so you can see where quality drops and by how much.

How many test cases do I need for an eval set?

There is no universal number, but a small set of realistic cases beats a large synthetic one. A few dozen carefully chosen examples can catch obvious regressions, while hundreds are needed to detect small improvements with any confidence. Cover your main use cases and known failure modes, add every production failure you find, and version the set so scores stay comparable. The question bank in this path discusses sizing in more depth.

Why evaluation is the hard part

Anyone can get an LLM feature to work once; proving it works reliably is what's hard — and what this path teaches. You'll build the vocabulary of evals (offline vs online, judges, rubrics), design an LLM-as-a-judge rubric that resists gaming, implement a real metric like token-level F1 by hand, then evaluate agents and a full production system rather than single calls.

The recurring lesson: a number you haven't validated against human judgement can mislead you with total confidence. This path is how you earn the right to trust your metrics — the skill that lets you actually improve an AI system instead of guessing.

← All learning paths