AI Evaluation
How you know an AI system actually works — LLM-as-judge, agent evals, and metrics like token-F1 — the half of AI engineering that separates demos from production.
Handbooks 5
51 LLM Evals Interview Questions
Golden sets, LLM-as-judge, regression testing, offline vs online evals, RAG evals, agent evals, red-teaming, and observability — demystified for interviews and production.
The Agent Evaluations Handbook
A self-contained handbook on evaluating AI agents — theory, interactive widgets, and practical guidance. Trajectory evals, tool-use scoring, LLM-as-judge, observability, and reliability for PMs, engineers, and founders.
The Evals in CI Handbook
Turn LLM quality into a regression gate, like unit tests for code. A golden set scored on every change, and why the gate needs both an aggregate threshold AND per-case no-regression — because a healthy average can hide a broken case (mean stays up while one case collapses). How to wire it into the PR pipeline, handle non-determinism with a tolerance band, and avoid a flaky gate. With worked math and runnable code.
Self-Improving Agents
The long one: how an agent gets better at its own job, end to end. COVERAGE vs the accuracy you can ship (a model usually already produces a right answer somewhere in its output; the missing piece is recognising it), unbiased pass@k, why the sampling curve is straight, the SELECTION bottleneck, building a VERIFIER and the two asymmetric ways it is wrong, feedback that is not a verifier ranked by independence, the five-predicate filter that decides what becomes training data, the SIX substrates a lesson can land in (context, memory, tools, scaffold, verifier, weights) and how to choose, STaR through PPO/GRPO/DPO and reward hacking, why a loop refit on its own output COLLAPSES toward its own habits, search as self-improvement, and measuring a time HORIZON rather than a score. 14 sections, 13 runnable Python blocks, 3 interactive models, 13 diagrams, 5 checkpoints and a running build log. Every figure carries the condition it was measured under.
Evals for Client Work
The most-cited hard skill in frontier-lab FDE job descriptions and the least taught anywhere: turning “it should be accurate” into a gate. Measure inter-expert agreement before you promise a number, build a golden set that survives contact, calibrate an LLM judge against human grades, wire the CI gate (including the per-case regression check), and hand the whole thing over.
Roadmaps 3
QA / SDET Roadmap
A step-by-step roadmap to become a QA engineer or SDET in 2026. From the test pyramid, unit and integration testing and test design through UI and API automation, frameworks, CI pipelines and flaky-test control to performance, security, testing AI systems and the SDET career. 18 stations across 3 tracks — Testing Foundations, Automation, Advanced Quality.
AI Safety Engineer Roadmap
A step-by-step roadmap to become an AI safety engineer in 2026. From alignment, evaluation, interpretability and threat modeling through prompt injection, guardrails, data poisoning, adversarial robustness and agent safety to policy, model documentation, responsible deployment and scalable oversight. 18 stations across 3 tracks — Foundations, Securing AI Systems, Governance & Frontier.
AI Red-Teamer Roadmap
A step-by-step roadmap to become an AI red-teamer in 2026 — the offensive side of AI security, explicitly distinct from the defensive AI-safety and classic security-engineer tracks. From the AI attack surface, threat modeling and red-team tooling through prompt injection, jailbreaks, data poisoning, model extraction and agent exploitation to injection defenses, gateways, supply-chain attacks, continuous red-teaming and responsible disclosure. 18 stations across 3 tracks — Foundations, Attack Techniques, Defense & Career.
System Designs 1
AI System Designs 9
Design LLM Eval & Observability
Build an LLM evaluation and observability pipeline. See how offline evals and online observability form two loops, how tracing every call is the foundation, how versioned golden datasets become your real benchmark, how programmatic/LLM-judge/human scorers combine, how a CI regression gate blocks regressions, how production monitoring catches drift, how failures feed back as fixtures — and why an unvalidated LLM judge silently corrupts every metric — through an interactive diagram.
Design a Model Registry (MLOps)
Build the platform that versions, evaluates, promotes and can instantly roll back trained ML models — the governance layer between "a model finished training" and "a model is safely serving production traffic." See why an artifact alone is meaningless without lineage, staged promotion through gated environments, automated evaluation gates that fail closed, shadow/canary deployment because offline metrics aren't sufficient, instant rollback via immutable versioning, and production drift monitoring.
Design a Data Labeling Platform
Build a human annotation platform like Scale AI or Labelbox — sourcing labels from people, not generating them programmatically. See why a single annotator's answer can't be trusted as ground truth, multi-annotator consensus, secretly-mixed gold-standard items that measure labeler accuracy in real time, skill-based task routing, active learning to prioritize which unlabeled examples matter most, and aligning labeler pay with measured quality.
Design a Prompt Management & A/B Testing Platform
Build the platform that versions, evaluates, and safely A/B tests prompts in a production LLM application — a much faster-moving artifact than model weights, often edited by non-engineers. See why hardcoded prompts force a full deploy for every tweak, externalizing prompts as versioned config, an offline eval gate before rollout, live A/B testing, guardrail metrics that must not regress, routine one-click rollback, and a review gate suited to non-engineer editors.
Design an AI Ad Creative Platform
Build a platform that generates ad creative at scale for advertisers, built on top of generative models rather than serving them directly. See why one hand-designed creative can't be tested at scale, programmatic variant generation, a brand-safety and ad-policy compliance gate that fails closed, rights and licensing tracking, dynamic multi-armed-bandit budget allocation, a performance feedback loop, and tiered generation cost control.
Design a Support Resolution Agent
Build an AI agent that triages, resolves, and escalates customer support tickets end-to-end. See why answering from general knowledge alone is dangerous when the real answer depends on account-specific state, grounding responses in real account/order data, confidence-based escalation instead of guessing, tiered auto/assisted/escalate resolution, seamless handoff context, outcome-based quality tracking, tone-aware de-escalation, and identity verification against social engineering.
Design a CI Test-Generation Agent
Build an agent that automatically generates tests for new and changed code in a CI pipeline. See why manual tests leave coverage gaps, coverage-gap analysis, generating tests against the real code, mutation testing to verify a test is actually meaningful, flaky-test detection before it poisons CI signal, why a human review gate is still required, and running expensive validation in a background lane.
Design Jev
Design a decision layer around a model like Jev, a model that never writes — 60 tokens at 20 ms is 1.2 seconds and $30,000 a day for a million tickets, against one pass at 0.1 seconds and $400. See why the input is a typed question sheet, why every answer comes out of one pass, what a calibrated probability is and how a squared penalty trains it, why a refund bar of 96% follows from a $50 mistake and a $2 review, how a cascade cuts the bill to $3,400, why state is fetched in code, and why a weekly audit of 200 labelled tickets catches a confident answer on changed traffic.
Design Cursor
Design an AI code editor like Cursor around the edit problem — retyping 4,000 tokens to change 30 is 80 seconds. See why a planner and an apply model are split, why the apply step rewrites the whole file instead of writing a diff (six line numbers at 2% is 11.4% wrong), how speculative edits use the old file as the draft for 110 passes and 2.2 seconds, how a Merkle tree re-embeds 60 chunks instead of 50,000, why Tab breaks even at exactly 25%, how checkpoints make an agent loop undoable, and why a hash check catches a right-looking edit made against a file that has moved.
Coding Challenges 1
Labs 1
Interactive Tools 2
LLM-as-Judge Rubric Builder
Define your evaluation criteria and a scoring scale, then generate a clean, copy-pasteable LLM-as-judge prompt you can drop into your eval pipeline — with the common pitfalls (position bias, verbosity bias, ties) called out. Turns eval theory into a prompt you can ship.
A/B Test Significance Calculator
An A/B test significance calculator using a two-proportion z-test. Enter the visitors and conversions for a control and a variant to get each conversion rate, the relative uplift, the z-score and two-tailed p-value, and a clear verdict on whether the result is statistically significant. Explains what significance does and does not tell you.
About AI evaluation
Shipping an AI feature is easy; knowing whether it's actually good is the hard part — and the part that separates demos from production. Unlike normal code, LLM output has no compiler and no exact-match "correct", so you need a real measurement discipline: golden datasets, scoring methods, and gates that catch a regression before your users do.
This topic covers the vocabulary and the machinery — offline vs online evals, LLM-as-a-judge and its biases, concrete metrics like token-level F1, and evaluating multi-step agents rather than single calls. The recurring lesson: a metric you haven't validated against human judgement can silently mislead you, so you calibrate the judge before you trust the number.