LLM Engineering
Building with large language models — prompting, decoding, inference serving and cost — from the handbooks down to the softmax that powers every token.
Handbooks 31
The Prompting Handbook
A friendly, hands-on field guide for everyday humans — learn the CRISP framework, spot bad prompts, practice with real recipes, play a drag-and-drop game, and test yourself with a quiz. No code required.
The Senior AI Engineer Interview Handbook
60 questions across architecture, production incidents, agentic systems, RAG, evals, cost, safety, and leadership — what staff-level AI interviewers actually probe for.
The LLM Serving Handbook
How to serve large language models fast and cheap — prefill vs decode, the KV cache, continuous batching and PagedAttention (vLLM), quantization, speculative decoding, and the latency-vs-throughput tradeoffs that decide your inference bill.
The Transformers Handbook
The architecture behind every LLM, built up from scratch — tokens and embeddings, positional encoding, self-attention (query/key/value), multi-head attention, the transformer block (FFN, residuals, layer norm), and how a decoder-only model generates text autoregressively.
The Fine-Tuning Handbook
When to fine-tune vs RAG vs prompting, full fine-tuning vs parameter-efficient methods, how LoRA and QLoRA make it cheap enough for a single GPU, why the data matters more than the method, catastrophic forgetting, and how to evaluate and decide.
The On-Device AI Handbook
Small models, everywhere: why 1–14B models got frontier-adjacent (distillation + textbook data), the napkin law of local inference (tokens/sec ≈ bandwidth ÷ bytes), quantization and the llama.cpp / GGUF / Ollama / MLX stack, the honest local-vs-cloud decision table, four hybrid patterns that ship, and the evals craft of choosing a small model that actually fits the device.
The GPU Fundamentals Handbook
The bottleneck is never where you think. SIMT and warps, the memory hierarchy from registers to HBM, arithmetic intensity and the roofline (compute-bound vs bandwidth-bound), where LLM workloads actually land, why FlashAttention was inevitable, the kernel stack from CUDA to Triton to torch.compile, and the multi-GPU interconnect story.
The Distributed Training Handbook
When the model doesn't fit, split it four ways. What actually eats training memory, data parallelism and all-reduce, ZeRO/FSDP sharding stages, tensor parallelism inside a layer, pipeline parallelism across layers (and the bubble micro-batching hides), 3D parallelism, the communication-overlap that decides MFU, and the memory tricks (mixed precision, gradient accumulation, activation checkpointing) you reach for first.
The AI Cost Engineering Handbook
AI spend is a sum over tokens: input-tokens × input-price + output-tokens × output-price, per million. Why output tokens dominate the bill (usually ~5× input), how prompt caching pays off from the very first reuse, and the full lever set — shorter outputs, caching, model routing, batching, context trimming — that cuts spend without cutting quality. With worked math and a runnable cost calculator.
The Model Routing Handbook
Send each request to the cheapest model that can handle it. How a cheap-smart cascade works, why you pay the cheap model on everything so the escalation rate is the key lever, the break-even math (r* = 1 − c_cheap/c_big), predictive routers vs cascades, and the confidently-wrong pitfall. With worked math and a runnable cost model.
The LLM Observability Handbook
Tracing an AI request as a tree of spans. Why total latency is the critical path — the max end time, not the sum of durations, because parallel spans overlap — while cost is the sum across every span, how to find the bottleneck span, and what to log for every LLM call (including percentiles over averages). With worked math and runnable code.
The Prompt Caching Handbook
How prompt caching actually works. Why caches key on the exact prefix and break at the first differing token, why cache-aware ordering (fixed content first, variable last) is the single biggest lever on hit rate, how TTL and cache breakpoints work, and the invisible mistakes (a timestamp up top, non-deterministic serialization) that silently kill caching. With worked math and runnable code.
The Local LLM Stack Handbook
Running LLMs locally with ollama and llama.cpp. The one equation that decides what fits — weight memory = params × bits per weight ÷ 8 — how quantization shrinks a 7B model from 14GB (fp16) to 3.5GB (4-bit) to fit a consumer GPU, why 4-bit is the sweet spot, and the local stack (GGUF, ollama, llama.cpp). With worked memory math and a runnable fits-in-VRAM calculator.
The Voice AI Real-Time Agents Handbook
Why a voice agent is a stopwatch, not a chat box. It's a three-stage pipeline — speech-to-text → LLM → text-to-speech — and the user waits for all three, so end-to-end latency is the sum of the stages. Human turn-taking breaks down past ~800ms, so the whole game is keeping that sum under the conversational budget: find the bottleneck (usually the LLM), shave the slowest stage first, cut network hops, and stream so the stages overlap. Plus endpointing, interruption handling, and why time-to-first-audio beats the naive sum. With worked math and runnable code.
The EU AI Act Handbook
The world's first comprehensive AI law, made legible for engineers. The key idea: it regulates AI by the risk of the use case, not the technology — the same model is unregulated in a spam filter and heavily regulated in a hiring tool. Every system sorts into one of four tiers: unacceptable (banned — social scoring, manipulation, most real-time public biometric ID), high-risk (hiring, credit, medical, education, law enforcement… → risk management, data governance, human oversight, conformity assessment, registration), limited-risk (transparency — chatbots disclose they're AI, deepfakes labelled), and minimal-risk (no obligations — the vast majority). Because tiering is a concrete rule, classification is deterministic. Plus the GPAI/foundation-model layer, what engineers should actually do, and the traps (optimistic self-classification, retrofitting compliance). With a worked classification model and runnable code.
The AI Product Engineering Handbook
A demo isn't a product. Getting an impressive LLM demo takes an afternoon; turning it into something you can charge for at scale is the brutal last mile where most AI features die. Product engineering rests on two numbers a demo never shows: unit economics (every request costs tokens — cost = in·p_in + out·p_out per million, which sets your gross margin and whether you're profitable) and the eval gate (ship a change only if its measured quality clears a bar AND its margin clears a bar — blocking cheap-but-wrong and great-but-unprofitable alike). Plus the reliability stack (evals, structured outputs, retries/fallbacks, caching, guardrails, observability) that keeps a flaky, pricey, non-deterministic model dependable in production. With worked math and a runnable cost/margin/ship-decision calculator.
LoRA vs Full Fine-Tuning
LoRA freezes the base model and trains a tiny low-rank adapter; full fine-tuning updates every weight. Why a low-rank update can work at all, memory and storage trade-offs, multi-task adapter serving, and when the extra cost of full fine-tuning is worth it.
LLM vs SLM
The choice is about deployment constraints, not just raw capability: LLMs run in the cloud with broad reasoning; SLMs run on-device with near-zero latency, cost, and full privacy. Why the capability gap is shrinking fast, and the model-routing pattern that uses both.
GPU vs TPU
GPUs are flexible general-purpose processors with the CUDA ecosystem behind them; TPUs are ASICs purpose-built around the systolic array for matmul, on Google Cloud only. Why that specialization is efficient exactly where it fits, and a liability the moment it doesn’t.
LLM API Pricing
Every major provider’s API pricing in one comparable table, per million tokens — Anthropic Claude, OpenAI GPT, Google Gemini — pulled from official docs on 2026-07-21, not aggregators. Batch and prompt-caching discounts, a cross-provider tier map, and a fully worked chatbot cost example.
Tokenizer Efficiency Benchmark
An original, reproducible benchmark of 9 real tokenizers across English prose, code, and non-English text — measuring characters-per-token and encode speed, with a fidelity check that catches the trap where a lossy tokenizer looks “efficient” only because it silently drops characters it can’t represent.
Test-Time Compute & Reasoning Models
Why letting a model think longer at inference — long chain-of-thought, sampling with self-consistency, best-of-N with a verifier, and search — can beat a much larger model on hard problems. How o1/o3- and DeepSeek R1-style reasoning models are trained to use a thinking budget, the train-time vs test-time scaling trade, and when spending the extra compute actually pays.
MoE vs Dense Models
Mixture of Experts vs dense LLMs, decided by which parameters fire per token: a dense model runs every parameter on every token; an MoE routes each token to a few expert sub-networks, holding far more total parameters but activating only a fraction. The compute win, the memory tax (all experts must sit in VRAM), load balancing, and when each wins.
Encoder vs Decoder Models
Encoder vs decoder Transformers, decided by which way attention flows: an encoder reads the whole input at once (bidirectional) to build representations — great for classification, NER and embeddings (BERT); a decoder generates left to right (causal) — great for chat, code and agents (GPT). Encoder-decoder models like T5 do both. When to reach for each, and why real systems chain them.
Quantization vs Distillation
Quantization vs distillation for shrinking LLMs, decided by what you change: quantization keeps the same model but stores its weights in fewer bits (FP16 → INT8/INT4); distillation trains a smaller student model to imitate a bigger teacher. One compresses the representation, the other the architecture — the quality-vs-size frontier of each, and why teams often distill then quantize.
Batch vs Real-Time Inference
Batch vs real-time (online) inference, decided by whether a human is waiting: batch processes many inputs together offline for maximum throughput and lowest cost per token; real-time answers one request at a time with low latency. GPU utilization, continuous batching, and why most AI products run both.
Mechanistic Interpretability
The engineer's guide to reading a model — interpretability as a debugger, not a séance. Why features live in superposition (not one per neuron), how sparse autoencoders pull monosemantic features back out, how attribution and circuit tracing find why a model did something (contribution of input i to output = the summed path weight W2·W1, validated causally by ablation), feature steering as a direct intervention, and an honest map of what you can debug today versus what is still research. Worked math plus a runnable 2-layer circuit tracer.
Diffusion LLMs
Text generation that isn't one token at a time. Why autoregressive decoding is latency-bound (N tokens = N sequential passes), how block diffusion denoises B tokens in K steps to emit B/K tokens per pass (⌈N/B⌉·K total passes, speedup B/K) for reported 1,000+ tokens/sec, the crucial image-vs-text split (continuous Gaussian noise over pixels vs a discrete masking corruption over tokens), and the speed-quality knob (fewer steps = faster but more parallel-commitment error). Worked math plus a runnable AR-vs-diffusion pass counter.
RL Environments Engineering
The build side of verifiable rewards — the reasoning models everyone celebrates are taught by the ENVIRONMENT, not the algorithm. Why RL learns only from reward variance p(1−p) (zero signal when every attempt passes or fails, peak at a 50% pass rate) so tasks must sit in the difficulty band, why a gameable verifier corrupts training (precision = correct ÷ all rewarded; every false positive is a lie the model learns), and how to engineer difficulty curricula and verifier soundness. The theory of reward hacking lives in the verifiable-rewards handbook; this is how to build around it. Worked math plus a runnable learning-signal and verifier-precision model.
Autoregressive vs Diffusion LLMs
Autoregressive vs diffusion language models, decided by how text is generated: autoregressive models emit one token at a time left-to-right, each conditioned on all previous; diffusion LLMs refine every position in parallel over a fixed number of denoising steps. Decode latency, KV cache, revisability, the quality gap, and where block diffusion fits.
vLLM vs TGI vs SGLang
Three open-source LLM serving engines that all do continuous batching but differ in their signature KV-cache trick: vLLM’s PagedAttention (removes fragmentation), SGLang’s RadixAttention (shares repeated prefixes), and TGI’s production-hardened Hugging Face integration. Where each wins, why "fastest" is a workload question, and how to choose.
Roadmaps 6
Prompt Engineering Roadmap
A visual transit-map roadmap from tokens and CRISP through chain-of-thought, RAG, and agent prompting to production monitoring. 18 stations across 3 tracks — interactive, free, your own pace.
AI Engineer Roadmap
A visual transit-map roadmap to become an AI engineer in 2026. From how LLMs work through embeddings, RAG, agents, and fine-tuning to evals, guardrails, inference serving, and observability. 18 stations across 3 tracks — Foundations, Build, Production.
MLOps Roadmap
A visual transit-map roadmap for MLOps in 2026. From reproducibility, data versioning, and experiment tracking through training pipelines and model serving to monitoring, drift, governance, and LLMOps. 18 stations across 3 tracks — Foundations, Pipelines, Operations.
Generative AI Roadmap
A visual transit-map roadmap for generative AI in 2026. From how text and images are generated through diffusion, multimodal, sampling, and fine-tuning to evaluating, securing, and shipping generative products. 18 stations across 3 tracks — Foundations, Techniques, Production.
Forward-Deployed Engineer Roadmap
A visual transit-map roadmap to the most in-demand AI role of 2026 — the engineer who ships AI where the customer lives. From rapid prototyping, enterprise RAG and integration glue through discovery, customer data wrangling, VPC deployment, security reviews and client evals to stakeholder craft, incidents, proving ROI, and scaling pilots into repeatable production. 18 stations across 3 tracks — The Craft, The Field, The Outcome.
ML Research Engineer Roadmap
A visual transit-map roadmap to become a machine-learning research engineer in 2026 — the person who builds and trains models, not just features on top of them. From the math and PyTorch, deep-learning fundamentals and reading papers through data pipelines, training dynamics, distributed training and scaling laws to post-training, evaluation, efficiency, the research process and the research-engineer career. 18 stations across 3 tracks — Research Foundations, Training at Scale, Frontier & the Craft.
AI System Designs 37
Design a Conversational AI
Build a production conversational AI system (think ChatGPT). See how the request path splits an inference gateway from the model servers, how the context window is assembled and token-budgeted, how conversation memory is stored and recalled, how tokens stream back over a persistent connection, and how guardrails gate every prompt and response.
Design an LLM Inference Server
Build an LLM inference serving system. See how a request queue absorbs spiky traffic, how the prefill/decode split and continuous batching keep GPUs full, how the KV cache and paged attention make each token cheap, how tensor sharding fits a giant model, and how autoscaling rides demand — all balancing latency against throughput,.
Design an LLM Gateway
Build an LLM gateway and model router. See why apps should call one provider-agnostic API instead of vendor SDKs, how adapters normalize every provider, how a capability-first router picks a model, how retries and failover survive a provider outage, how per-tenant limits and budgets isolate a shared quota, how caching cuts cost and latency — and how cost-only routing silently wrecks quality.
Design an LLM Cache
Build a prompt & response caching layer for LLMs. See why repeated and near-duplicate prompts should skip the model, how an exact-match cache keys on the normalized request, how a semantic cache reuses answers above a tuned similarity threshold, how prompt-prefix (KV) reuse cheapens even misses, how TTL and invalidation keep it fresh — and how a loose threshold silently serves wrong answers.
Design a Fine-Tuning Pipeline
Build an LLM fine-tuning and training pipeline. See how datasets are curated (the real work), how a pretrained base is adapted with LoRA/PEFT, how the distributed training loop is monitored, how a held-out eval gate decides promotion, how a versioned model registry enables rollback, how canary deploys ship safely, and how the production data flywheel compounds — plus why eval-set contamination inflates metrics silently — through an interactive diagram.
Design Content Moderation
Build a content-moderation pipeline (text + image). See how a staged funnel hash-matches known-bad content, how text and multimodal classifiers emit per-category scores, how OCR closes the text-in-image loophole, how a policy engine maps scores to graduated actions, how a human review queue handles the uncertain middle, how appeals and a retraining loop fight adversarial evasion — and why fully trusting the classifier fails — through an interactive diagram.
Design an AI Coding Assistant
Build an AI coding assistant (like Copilot or Cursor). See how inline completion / tab prediction meets a sub-second latency budget, how context is assembled with fill-in-the-middle, how repo-aware retrieval grounds completions in the codebase, how a code-specialized model streams suggestions, how debounce/cancel/cache win the milliseconds, how acceptance-rate telemetry measures quality, how an agentic chat mode handles multi-file edits, and why starving the context yields confident wrong code — through an interactive diagram.
Design Multi-Agent Orchestration
Build a multi-agent orchestration / workflow engine. See how an orchestrator decomposes a goal across specialist agents, when NOT to go multi-agent, how shared state coordinates them, how a durable workflow engine checkpoints and resumes, how typed handoffs stop telephone-game degradation, how fan-out/fan-in parallelizes independent work, how budgets and termination prevent runaways, and why uncapped agents loop and burn unbounded cost — through an interactive diagram.
Design LLM Eval & Observability
Build an LLM evaluation and observability pipeline. See how offline evals and online observability form two loops, how tracing every call is the foundation, how versioned golden datasets become your real benchmark, how programmatic/LLM-judge/human scorers combine, how a CI regression gate blocks regressions, how production monitoring catches drift, how failures feed back as fixtures — and why an unvalidated LLM judge silently corrupts every metric — through an interactive diagram.
Design an LLM Guardrails System
Build an LLM guardrails / safety-filter system. See how a safety pipeline wraps every call, how input guardrails screen PII, prompt-injection and scope, why injection needs defense-in-depth, how output guardrails check toxicity, PII, schema and groundedness, how violations are handled gracefully, how cheap-first layering keeps it affordable, why both over- and under-blocking fail — and how trusting untrusted input lets injection through — through an interactive diagram.
Design a Text-to-Image Service
Build a text-to-image generation service like Midjourney, DALL·E or hosted Stable Diffusion. See why synchronous generation fails, how the async job pattern accepts fast and works later, how a queue absorbs bursts against fixed GPU capacity, what the diffusion denoising loop actually does, how batching and autoscaling keep the GPU fleet cost-effective, how object storage + CDN deliver images, and why moderation is required on both the prompt and the image.
Design a Speech-to-Text Service
Build a speech-to-text / ASR service like Whisper-at-scale or Deepgram. See why batch and streaming transcription have opposite constraints, how a gateway splits the paths over one GPU fleet, how VAD and overlapping chunks feed a fixed model window, what the ASR inference pipeline does, how batching and autoscaling keep GPUs economical, and how post-processing (punctuation, timestamps, diarization) turns raw tokens into a usable transcript.
Design a Text-to-Speech Service
Build a text-to-speech / voice synthesis service like ElevenLabs or Play.ht. See why natural TTS is a pipeline (text normalization → acoustic model → vocoder), how batch and streaming differ, why time-to-first-audio is the metric for voice agents, how voices and cloning work, how caching common phrases skips the GPU, how GPU scaling protects the latency budget, and why cloning demands consent and watermarking.
Design a Realtime Voice Agent
Build a realtime voice AI agent like ChatGPT voice mode or an AI phone agent. See why a sequential STT→LLM→TTS pipeline feels robotic, how streaming and pipelining the three stages cut per-turn latency, how turn-taking (endpointing + barge-in) makes it feel human, how the interruptible orchestrator manages state and clean cancellation, where the latency budget goes, and how tools and transport (WebRTC/SIP) fit in.
Design an AI Answer Engine
Build an AI answer engine like Perplexity. See why a bare LLM is stale and hallucinates with no sources, how retrieve-then-generate (RAG over the live web) grounds answers, how a query planner rewrites and decomposes questions, how search + fetch + rerank build a tight evidence set, how grounded generation cites every claim, how citation verification catches unsupported statements, and how to defend against prompt injection from web content.
Design an Embeddings Service
Build an embeddings service that powers semantic search, RAG and recommendations. See why embedding needs a shared service, how online (low-latency query) and batch (millions of docs) modes differ, what the model outputs, how a batched GPU pipeline indexes a corpus, the critical rule that query and corpus must share the same model version (so a model change re-embeds everything), and how caching by (text, version) saves cost.
Design a GraphRAG System
Build a GraphRAG system — knowledge-graph-augmented retrieval. See why vanilla vector RAG fails on multi-hop and global questions, how a knowledge graph of entities and relationships lets retrieval traverse connections, how LLM extraction builds the graph, how community detection and hierarchical summaries enable global questions, how local vs global query modes work, and why hybrid graph+vector retrieval is the strongest form.
Design a Code Execution Sandbox
Build a secure code execution sandbox for untrusted, LLM-generated (or user-submitted) code — the runtime behind code agents, online judges and notebooks. See why running code directly is a full compromise, how containers and microVMs isolate it, how resource limits stop hangs and fork bombs, the submit→isolate→capture pipeline, why sandboxes must be ephemeral, how warm pools keep it fast, and how egress control prevents exfiltration.
Design an LLM Router
Build an LLM router that sends each request to the cheapest model that can handle it. See why one-model-for-all overpays on the easy majority and fails the hard minority, how a fast difficulty classifier provides the routing signal, how tiered routing with a cheap-first cascade survives classifier errors, how semantic routing to specialists is cheaper and better, how caching and multi-provider fallback add resilience, and why the quality bar comes before cost.
Design a Batch Inference System
Build an offline batch LLM inference system that grinds millions of prompts at maximum throughput per dollar. See why looping the online API fails, how the async job model works, how huge and continuous batching saturate the GPU, how a sharded restartable pipeline survives failures, how length bucketing and prefix caching squeeze each batch, and how checkpointing unlocks cheap spot GPUs and batch-discount pricing.
Design a Synthetic Data Pipeline
Build a synthetic data pipeline that uses LLMs to generate training and eval data. See why naive "generate 10,000 examples" gives low-diversity, error-prone data, how a strong teacher model generates candidates, how diversity steering avoids mode collapse, how quality filtering and verification (execute code, ground facts) keep only good examples, how the data powers distillation/augmentation/evals, and how to avoid model collapse and contamination.
Design an RLHF Pipeline
Build an RLHF alignment pipeline. See why a pretrained base model is capable but unaligned, how supervised fine-tuning teaches instruction-following, why preference comparisons beat demonstrations, how a reward model turns human comparisons into a dense score, how PPO optimizes the policy with a KL constraint to prevent reward hacking, the DPO and RLAIF alternatives, and the reward-hacking, overoptimization and labeler-quality traps.
Design a Document AI Pipeline
Build a document AI / intelligent document processing pipeline that turns PDFs, scans and forms into structured data. See why plain OCR gives characters but loses structure, how preprocessing and OCR produce located text, how layout analysis recovers 2D structure (tables, key-value pairs), how templates vs ML vs VLM extraction produce schema JSON, how validation and confidence catch misreads and hallucinated fields, and how human-in-the-loop review keeps it accurate.
Design ChatGPT
Design ChatGPT itself — a chat product serving hundreds of millions of weekly users and billions of prompts a day. See why the serving path must be stateless, how conversations shard by id, how the context is rebuilt every turn under a token budget, how a model router becomes the #1 cost lever, how prefix caching kills redundant prefill, how continuous batching squeezes the GPU fleet, and the load-shedding playbook for a capacity crunch.
Design an AI Meeting Notetaker
Build an AI meeting notetaker. See how a capture bot joins calls with per-speaker audio, why consent is architecture rather than a checkbox, how streaming ASR and diarization turn chaos into an attributed transcript, why extracted decisions and action items must cite timestamped segments, how meeting memory answers questions across months of calls, and how artifacts land in the tools where work happens — through an interactive diagram where you can unleash crosstalk.
Design a Text-to-Video System
Build a Sora-style text-to-video service. See why minutes-long renders force an async job spine, how safety gates both prompts and pixels, why an LLM expands prompts toward the training distribution, how a GPU scheduler runs queue economics with tiers and preemption, how diffusion transformers denoise spacetime latents jointly for coherent motion, why C2PA provenance and watermarking ship on every render, and how draft-then-final rendering makes iteration affordable — through an interactive diagram with a viral-surge chaos mode.
Design an Adaptive AI Tutor
Build an adaptive tutoring system like Khanmigo or Duolingo's AI tutor. See why a fixed question order fails every learner, how a per-skill mastery model picks the next problem, why grading routes structured answers to a deterministic checker and reserves the LLM for open-ended judgment, how the mastery feedback loop closes, why explanations are grounded in a curriculum knowledge base instead of freely generated, how a Socratic hint ladder avoids just giving away the answer, and how a long-term learner profile drives spaced repetition.
Design an AI Code Review Bot
Build an automated PR review bot like CodeRabbit or Graphite. See why cheap deterministic linters run before any LLM call, how review is scoped to the diff plus just enough context, how a repo embedding index (RAG over the codebase) supplies cross-file context the diff alone can't show, why every finding is confidence-gated before posting, how a comment ledger prevents re-pushes from spamming old feedback, how secrets are redacted before they ever reach the LLM, and how developer reactions tune down false positives over time.
Design a Model Registry (MLOps)
Build the platform that versions, evaluates, promotes and can instantly roll back trained ML models — the governance layer between "a model finished training" and "a model is safely serving production traffic." See why an artifact alone is meaningless without lineage, staged promotion through gated environments, automated evaluation gates that fail closed, shadow/canary deployment because offline metrics aren't sufficient, instant rollback via immutable versioning, and production drift monitoring.
Design a GPU Cluster Scheduler
Build a scheduler for a shared pool of expensive GPUs across many teams. See why naive FIFO scheduling starves large distributed jobs and fragments capacity, gang scheduling for all-or-nothing distributed training, topology-aware placement, priority and preemption for latency-sensitive inference versus long-running batch training, fair-share quotas across competing teams, checkpoint-and-resume, and reconciling the scheduler's bookkeeping against real cluster state.
Design a Prompt Management & A/B Testing Platform
Build the platform that versions, evaluates, and safely A/B tests prompts in a production LLM application — a much faster-moving artifact than model weights, often edited by non-engineers. See why hardcoded prompts force a full deploy for every tweak, externalizing prompts as versioned config, an offline eval gate before rollout, live A/B testing, guardrail metrics that must not regress, routine one-click rollback, and a review gate suited to non-engineer editors.
Design a Multi-Tenant AI SaaS Platform
Build an AI product serving many separate customer organizations from shared infrastructure. See why unscoped shared retrieval risks cross-tenant data leaks, enforcing isolation at the data layer, per-tenant customization via configuration, noisy-neighbor prevention with per-tenant quotas, per-tenant cost attribution, data-residency routing, automated onboarding, and defense-in-depth isolation that survives a bug in any single layer.
Design an Autonomous Email Agent
Build an AI agent that triages, drafts, and sends email on a user's behalf, where sending is often irreversible. See why auto-sending everything is dangerous, a human-approval gate scoped to action reversibility, priority triage, context assembly from thread history, tone/style matching, defending against indirect prompt injection hidden in received email, and a full audit trail.
Design an Edge Inference Fleet
Build the platform that deploys and manages ML inference across a large fleet of edge devices — phones, IoT sensors, kiosks, cars. See why cloud-only inference fails at fleet scale, model compression for severe resource budgets, offline-capable on-device inference, staged rollout that's harder than server rollback, bandwidth-conscious delta updates, telemetry from devices you don't control, graceful degradation, and devices that self-heal their model version on reconnect.
Design an AI Video Dubbing System
Build a system that translates and re-synthesizes speech in existing video, synchronized to lip movements — an offline batch problem with its own timing and ethical constraints. See why naive translate-and-overlay drifts out of sync, timing-constrained translation, higher-quality offline voice cloning, the audio-vs-video lip-sync trade-off, cross-segment consistency, consent for voice likeness, review gates scaled by content stakes, and background audio preservation.
Deploy an LLM in a Customer’s Environment
Deploy a large language model inside a customer’s own environment — the constraint a forward deployed engineer meets when a regulated customer won’t send data to a public API. See why a hosted endpoint is a non-starter, right-sizing the model and precision to the fixed hardware they actually own, serving efficiently on limited GPUs, running with no network egress (air-gapped, no phone-home), shipping model and security updates as signed bundles into a locked-down environment, and getting observability out without exfiltrating customer data.
Design an RL Environment Farm
Build the infrastructure that trains reasoning models — the environment-and-verifier side of RL with verifiable rewards, not the preference-tuning of RLHF. See why running rollouts inline in the trainer is unsafe and unscalable, sandboxed rollout workers that safely execute untrusted model-generated code, a scheduler that fans rollouts across a fleet and distributes the policy, a SOUND verifier (a gameable one makes the model learn the exploit — reward hacking), a task bank that serves the productive difficulty band (a curriculum, because learning needs reward variance), and a trajectory store that closes the loop — with reward-hacking and sandbox-escape chaos.
Paper Breakdowns 51
Attention Is All You Need
The paper that introduced the Transformer — the architecture underneath GPT, BERT, Claude, and Gemini. Self-attention, Query/Key/Value, multi-head attention, positional encoding, and the real BLEU results, explained in depth with analogies and animated visuals, not just equations.
BERT
The paper that made "pre-train once, fine-tune anywhere" the default for NLP. Bidirectional context, masked language modeling, next-sentence prediction, the [CLS]/[SEP] tokens, and the GLUE sweep — explained with analogies, not just equations.
Chain-of-Thought Prompting
Show a model a few worked examples with their reasoning and it starts solving problems it used to fail. Why reasoning emerges only at scale, the GSM8K jump, zero-shot "Let's think step by step", and how CoT seeded modern reasoning models.
LoRA
The paper that made fine-tuning affordable. Freeze the giant pretrained model and train two tiny low-rank matrices instead — up to 10,000x fewer trainable parameters, no extra inference latency, and swappable per-task adapters. The road to QLoRA and PEFT.
Direct Preference Optimization
RLHF without the RL. Align a model to human preference pairs with one simple classification-style loss — no separate reward model, no unstable reinforcement learning. Why "your LM is secretly a reward model", and how DPO became the default alignment method.
Chinchilla
The compute-optimal scaling law that said models were too big. Parameters and training tokens should scale together — ~20 tokens per parameter — so a 70B model trained on 1.4T tokens beat the 280B giants. Why every model since trains on trillions of tokens.
FlashAttention
Attention was slow for the wrong reason. By treating it as a memory-movement problem — tiling, online softmax, keeping work in fast SRAM instead of slow HBM — FlashAttention makes exact attention several times faster and cuts its memory from quadratic to linear, unlocking long context windows.
Mixture of Experts
How models got huge without getting slow. Many expert sub-networks plus a gating network that routes each token to just a few — so a model holds hundreds of billions of parameters but only activates a fraction per token. Gating, top-k routing, load balancing, and the road to Switch Transformer and Mixtral.
GPT-3
The paper that made models learn from the prompt. Scale a Transformer decoder to 175 billion parameters and a new behavior emerges — in-context learning: teach it a task with a few examples in the prompt, no fine-tuning, no weight updates. Zero-, one-, and few-shot prompting explained.
InstructGPT
How RLHF taught models to follow instructions — the paper behind ChatGPT. The three-step recipe: supervised fine-tuning, a reward model learned from human rankings, and PPO reinforcement learning. A 1.3B aligned model beat 175B GPT-3 on human preference — alignment over scale.
Llama 2
The paper that opened up the LLM. Meta released a strong family of open-weight models — 7B, 13B, 70B — with chat alignment included. Pretraining on 2T tokens, grouped-query attention, and Llama 2-Chat's RLHF with separate helpfulness and safety reward models — the release that seeded the open-model ecosystem.
Scaling Laws
The physics of making models bigger. A language model's loss falls as a smooth power law in model size, dataset size, and compute — so performance is predictable before you train. Compute-optimal training, the forecast that justified GPT-3, and how Chinchilla later corrected the recipe.
T5
The model that turned every NLP task into text-to-text. One encoder-decoder Transformer, one loss, one format: translation, summarization, classification, and Q&A all become text-in, text-out. The C4 dataset, span-corruption pre-training, and a rigorous study of what drives transfer learning.
Seq2Seq
Teaching neural nets to translate. An encoder LSTM reads an input sentence into a single fixed vector; a decoder LSTM writes the translation out. The architecture that made neural machine translation work — and whose fixed-vector bottleneck directly motivated the attention mechanism.
DeepSeek-R1
How a model learned to reason from reward alone. Reward only correct final answers and step-by-step chain-of-thought emerges on its own — including a spontaneous "aha moment" of self-correction. R1-Zero's pure RL, cold-start data, and cheap distillation of reasoning into small open models.
Toolformer
How a language model taught itself to use APIs. Toolformer learns, self-supervised, to call a calculator, search, QA, translation, and calendar — inserting API calls into text and keeping only the calls that make its next-word prediction better. The recipe that seeded modern tool-using AI agents.
GRPO (DeepSeekMath)
The RL algorithm behind DeepSeek-R1. PPO needs a critic as large as the model itself; GRPO deletes it — sample a group of answers, score them, and reward whoever beats the group average. Half the memory, none of the critic's instability, and the engine that later made reasoning emerge from reward alone.
Titans
Models that memorize at test time. A neural long-term memory keeps learning during inference — writing what surprises it into its own weights, with momentum and adaptive forgetting — so attention handles the recent window while the memory holds the far past. Beats Transformers and Mamba on 2M+ token recall.
Constitutional AI
Alignment from principles, not labels. Write the values down as a constitution, have the model critique and revise its own outputs against them, then run RL on AI-judged preferences (RLAIF) — harmlessness that scales with compute instead of crowds, and refuses without stonewalling. The backbone of Claude's alignment.
Self-Consistency
Sample many diverse reasoning paths, marginalize out the reasoning, majority-vote the answer. Correct answers are attractors — many routes converge on them while scattered errors disagree. +17.9 points on GSM8K from a pure decoding change, and the first clean proof that inference compute is a quality dial.
Tree of Thoughts
When LLMs learn to search. Frame problem-solving as a tree of intermediate "thoughts": propose candidates, self-evaluate their promise, expand with BFS/DFS, backtrack from dead ends. Game of 24 went 4% → 74% with the same GPT-4 — the inference procedure, not the model, was the ceiling.
QLoRA
Fine-tune a 65B model on one GPU. Freeze the base in 4-bit NF4 (levels at the quantiles of a normal distribution), quantize the quantization constants, page optimizer spikes to CPU, and train 16-bit LoRA adapters on top — matching full 16-bit fine-tuning quality. Guanaco hit 99.3% of ChatGPT after 24h on a single card.
DeepSeek-V3
Frontier performance at a tenth of the cost. A 671B MoE with 37B active, multi-head latent attention shrinking the KV cache ~10×, auxiliary-loss-free expert balancing, multi-token prediction, and FP8 training with zero loss spikes — 14.8T tokens for ≈$5.6M of compute. The paper that reset the industry’s cost assumptions.
Kimi K2
A trillion parameters built for agents. A 1T-param open MoE (32B active) trained through 15.5T tokens with ZERO loss spikes via MuonClip (Muon + QK-clip), plus large-scale synthesis of tool-use episodes so agentic behavior is pretrained, not bolted on. The moment open weights led on agent workloads.
LayerNorm & RMSNorm
The normalization inside every transformer. LayerNorm normalizes across one example’s features (batch-independent, unlike batch norm); RMSNorm drops the mean-centering to go faster. The exact invariances that separate them, and the tidy case where they’re mathematically identical — worked math plus runnable code.
Speculative Decoding
Make a big model generate 2–3× faster with zero change to its output. A tiny draft model guesses several tokens ahead; the target verifies them all in one parallel pass and keeps the correct prefix. Why decoding is memory-bound, why the output is provably identical, and the expected-tokens math with its optimal draft length — worked math plus runnable code.
RoPE (RoFormer)
The position scheme inside LLaMA, GPT-NeoX, and most modern LLMs. Instead of adding a position vector, RoPE rotates query and key by an angle set by their position — and because rotations add, the attention score depends only on relative distance. The geometry, the per-dimension frequencies, and long-context scaling — worked math plus runnable code.
Switch Transformer
The mixture-of-experts model that scaled to 1.6 trillion parameters by routing each token to a single expert. Why top-1 routing simplifies MoE, the load-balancing auxiliary loss that keeps experts evenly used, expert capacity and token dropping, and how sparse activation decouples parameters from compute per token — worked math plus runnable code.
PagedAttention (vLLM)
The idea that made LLM serving several times cheaper: store the KV cache in fixed-size blocks like an OS pages memory, instead of reserving each request’s maximum length. Why contiguous reservation wastes ~95% of the cache, how paging caps the waste to one block, copy-on-write prefix sharing, and the throughput vLLM unlocks — worked math plus runnable code.
Longformer
The attention pattern that let transformers read long documents. Full self-attention is quadratic, so it caps out around 512 tokens; Longformer’s sliding window makes it linear in length, with dilation and depth for reach and a few global tokens for the big picture. The sparsity math and receptive-field growth — worked math plus runnable code.
Medusa
Speculative decoding without a separate draft model. Medusa bolts extra prediction heads onto the model, each guessing a token further ahead, and verifies many candidate continuations in one pass via tree attention. Why head accuracy fades with distance, the tokens-per-pass math, and the trade-offs vs draft-model speculation — worked math plus runnable code.
Let's Verify Step by Step
Reward every reasoning step, not just the final answer. Process reward models beat outcome reward models at judging hard math — they catch right-answer/wrong-reasoning, localize the failing step, and rerank many candidate solutions to pick sound ones. The process-vs-outcome math and the PRM800K dataset — worked math plus runnable code.
Textbooks Are All You Need (Phi)
A 1.3B model trained on curated, textbook-quality data rivaled models 10× larger. Data quality, not just scale, drives capability — a small clean dataset can carry more effective signal (tokens × quality) than a giant noisy one, and the small model is far cheaper to run. The effective-signal argument, compute math, and caveats — worked math plus runnable code.
Transformer-XL
The transformer that broke the fixed-context limit. Segment-level recurrence caches the previous segment's hidden states so context grows with depth, and relative positional encoding makes the recurrence coherent. Why vanilla transformers fragment context, how the cache reaches thousands of tokens back, and the ~1,800× evaluation speedup — worked math plus runnable code.
Ring Attention
Context that scales with the number of devices. Shard the sequence across a ring, rotate the key/value blocks around it so every query sees every key, and per-device memory stays O(N/P) — with communication hidden behind computation. Exact dense attention, no approximation, pushing context into the millions. The rotation and overlap math — worked math plus runnable code.
STaR
A model teaching itself to reason. STaR generates chain-of-thought rationales, keeps only the ones that reach the correct answer, fine-tunes on them, and repeats — bootstrapping reasoning from question-answer pairs alone. Rationalization (hinting the answer) rescues hard problems. The self-improvement loop, foreshadowing RL-trained reasoning models — worked math plus runnable code.
Codex
The GPT-on-code model behind GitHub Copilot — and the evaluation it standardized. Codex introduced HumanEval and pass@k: grade generated programs by whether they pass the unit tests, and measure the chance at least one of k samples works. The unbiased pass@k estimator (1 − C(n−c,k)/C(n,k)) and why sampling helps — worked math plus runnable code.
SWE-bench & SWE-agent
Can a model do real software engineering? SWE-bench grades models on resolving actual GitHub issues — the patch must apply, make the failing tests pass, and break no passing tests. SWE-agent gives the model an Agent-Computer Interface (search/open/edit/run) built for the model, not humans. The strict resolution rule and why the interface matters — worked math plus runnable code.
AlphaCode
Median-human competitive programming by scale plus selection. AlphaCode samples millions of candidate programs, filters out those that fail the example tests, clusters the survivors by behavior, and submits a few representatives from the largest clusters — consensus as majority vote over code. The sample-filter-cluster pipeline and why it works — worked math plus runnable code.
Llama 3
A herd of open dense transformers trained far past the Chinchilla point. Llama 3 deliberately over-trains its smaller models on ~15 trillion tokens — nearly 100× the compute-optimal count — because a model is trained once but served billions of times, so inference, not training, dominates the lifetime bill. The training-optimal versus inference-optimal trade-off, with worked math and runnable code.
Gemini 1.5
A million-token context window with near-perfect needle-in-a-haystack recall. Gemini 1.5, a sparse mixture-of-experts model, can retrieve a fact hidden anywhere in up to 1M (research: 10M) tokens — because its usable attention span covers the whole window (W ≥ L), where a limited window drops every early needle. Why long context is hard, why it is expensive, and how to test it — worked recall condition plus runnable code.
PaLM
A 540-billion-parameter dense model trained with Pathways that made emergent abilities concrete. Some capabilities stay near chance until scale crosses a threshold, then jump sharply — because a task needing k sequential correct steps scores p^k, so a smooth per-step improvement becomes a sudden task-level breakthrough. Compositional emergence, why harder tasks emerge later, and what Pathways enabled — worked math plus runnable code.
Qwen3
An open model family that unifies a thinking mode (chain-of-thought) and a non-thinking mode (direct answer) in one model, with a controllable thinking budget. Since thinking gains saturate and help hard queries far more than easy ones, gating reasoning on difficulty — skip easy, spend the budget on hard — maximizes accuracy per token, beating always-think and never-think. The test-time-compute trade-off as a dial — worked math plus runnable code.
GPT-4 Technical Report
The report is famous for what it withholds, but its key methodological claim is predictable scaling: OpenAI predicted GPT-4's final loss from models trained with up to ~1000× less compute by fitting a power law and extrapolating. Why loss falls as a smooth power law you can fit on cheap small runs and extend across a huge compute gap — turning a giant, risky training run into an engineering forecast — and where capability prediction still breaks. Worked math plus runnable code.
OpenAI o1
The reasoning model trained with reinforcement learning to think before it answers — and the discovery of a second scaling axis. For a fixed model, accuracy rises roughly log-linearly with test-time (inference) compute: each ×10 in thinking buys a fixed accuracy step, so reaching a higher accuracy costs a multiplicative jump in compute. Reasoning as search under a budget, the inference-scaling law, and its exponential price — worked math plus runnable code.
DeepSeek-V4
The efficiency sequel to V3, aimed at the real cost of long context. A ~1.6T-parameter MoE (only ~49B active per token) that serves a 1M-token window at roughly a tenth of V3.2's KV cache. The mechanism is one idea on two faces — HYBRID sparse attention: compress each token's keys/values into a small latent (memory per token: ratio = d_c/width, context-invariant), then SELECT only the top-k blocks of size b to score against, freezing the scored-key count at k·b once n>k·b while dense attention keeps paying O(n²). The two discounts multiply: FLOPs ratio = (k·b/n)·(d_c/width). Reasoning is post-trained with ON-POLICY DISTILLATION — the student generates its own trajectories and a stronger teacher corrects those exact outputs — replacing R1's large-scale RL loop with a denser, more stable signal. Worked math plus runnable code.
Kimi K3
The sequel to Kimi K2, and a different bet: attack decode SPEED, not KV memory. A ~2.8T open MoE with a 1M-token context — reported as the largest open model to date — built on KIMI DELTA ATTENTION (KDA), a linear-attention delta rule. Softmax attention re-reads the whole KV cache every decode step (O(n) per token, so generation slows as context grows); KDA folds the past into ONE fixed-size recurrent state S (~d×d) and reads o=S·q in constant time regardless of length. The DELTA rule (S ← S − β·(S·k − v)·kᵀ) makes each write CORRECTIVE — overwrite a key's old value instead of accumulating it — which is what plain linear attention gets wrong (write (k,a) then (k,b): delta reads b, linear reads a+b, smeared). ATTENTION RESIDUALS keep a minority of full-attention layers for the sharp exact recall a fixed state blurs — a hybrid, not a wholesale swap. Worked math plus runnable code.
Engram
The memory half of the long-context problem. Softmax attention answers a query by spreading a fixed probability budget over ALL tokens, so as the haystack grows the needle dilutes and recall degrades — you pay O(n) AND lose accuracy. Engram bolts a KEY-ADDRESSED memory onto the model: write a fact under a key, retrieve it in constant time O(1) regardless of context length, no scan and no dilution (a Python dict IS the abstraction). It is CONDITIONAL — a lightweight gate decides whether a position needs a long-range recall, so local tokens pay nothing and only recall tokens incur one O(1) lookup. Because retrieval is by identity not by out-competing every other token, needle-in-a-haystack recall stops falling with length (reported ~84.2→97). Underpins DeepSeek-V4 (cheap context that stays accurate); same explicit-memory family as Titans and MemGPT, baked into the model not bolted on by a harness. Worked math plus runnable code.
Qwen4-Coder
An open, Apache-2.0 coding model that reportedly hits ~82% on SWE-Verified while running on a MacBook — the first open, Mac-runnable model past 80% there. The enabler is the mixture-of-experts SHAPE, not a bigger model: ~32B total parameters but only ~3B active per token, which splits the two costs everyone conflates. STORAGE is set by total params (all experts must be RAM-resident since the router may pick any) — 32B at 4-bit ≈ 16GB, fits a 24GB Mac (fp16 would be 64GB and fail); quantization is what makes it a laptop model. SPEED is set by ACTIVE params (decode is bandwidth-bound and reads only the routed ~3B experts) — so a 32B model decodes at 3B throughput. The mental model: "does it fit?" is answered by total×bits÷8; "is it fast enough?" by active params vs bandwidth — dense models tie these together, MoE pulls them apart. Sequel to Qwen3; validates the local-first coding-agent stack. Worked math plus runnable code.
GPT-5.6 System Card
A modern system card is really a COMPUTE-ALLOCATION POLICY, not one model. TIERED ROUTING (reported Sol/Terra/Luna) sorts each query by difficulty — cheap fast tier by default, escalate the hard cases (the cascade cost math lives in the model-routing handbook — linked, not re-derived). ULTRA MODE spends test-time compute in PARALLEL: run several independent agents (reported 4) on the hardest task and a VERIFIER keeps any that solves, so the solve rate is 1−(1−p)^N. The fresh angle = why FOUR, not forty: the marginal value of the N-th agent is Δ(N)=(1−p)^(N−1)·p — geometric decay (p=0.6: +0.60/+0.24/+0.10/+0.04) while cost is linear in N, so there's a knee. Runnable proves best-of-N rises, marginal=(1−p)^(N−1)·p, diminishing returns, 4th≪1st, value/cost decreasing, and finds the knee. PREPAREDNESS: safety case must cover the STRONGEST config (top tier + full ultra), not the cheap default. Cross-links o1-system-card (scaling curve) + model-routing/ai-cost-engineering + self-consistency. Worked math plus runnable code.
Concept Circuits
The foundation of mechanistic interpretability — how to read a model. Neurons are POLYSEMANTIC (one unit fires for green AND legal text AND Python) because the model stores concepts as DIRECTIONS spread across many neurons, not one-per-neuron. SUPERPOSITION: a d-dim space holds only d orthogonal directions but VASTLY more near-orthogonal ones, so a model packs in more features than dimensions and accepts small interference (interference(i,j)=feature_i·feature_j: orthogonal=0, superposed small-nonzero; 3 unit vecs at 120° in 2-D → pairwise dot −0.5). It works because features are SPARSE — few active at once, so collisions rarely bite and a lone active feature reads back cleanly. ATTRIBUTION GRAPHS trace which features caused an output (contribution = output-direction · feature-direction) and chain them across layers into a CIRCUIT. Runnable proves orthonormal self=1/cross=0, 3 features in 2-D, unit length preserved, interference bounded 0.5, clean sparse readout. First interp content sitewide; pairs the coming mech-interp-for-engineers handbook. Worked math plus runnable code.
Coding Challenges 11
Softmax
The function at the end of every classifier and language model: turn raw scores (logits) into a probability distribution. Implement the numerically stable version so big logits do not overflow. Solve it in Python or TypeScript.
Single-Head Attention
The operation at the heart of every Transformer: scaled dot-product attention. Given queries, keys, and values, let each query pull a weighted blend of the values — softmax(QKᵀ/√d)·V — with a numerically stable softmax and no numpy, just the math. Solve it in Python or TypeScript.
Layer Normalization
The stabilizer wrapped around every Transformer sub-layer: re-center and re-scale a vector to mean 0 and variance 1, then let learned gamma and beta stretch and shift it back — y = gamma·(x−mean)/√(var+eps) + beta. Keeps deep nets trainable. Solve it in Python or TypeScript.
Cross-Entropy Loss
The loss that trains almost every classifier and language model. It measures how surprised the model was by the right answer — high probability on the true class → near 0, confidently wrong → explodes. Return -log(p[target]), eps-guarded. Solve it in Python or TypeScript.
TF-IDF
The scoring that ran search for decades — and still seeds hybrid retrieval today. Reward a word frequent in one document but rare across the corpus, shrug off words that appear everywhere: tf·log(N/df). Solve it in Python or TypeScript.
BPE Merge Step
One step of how every tokenizer vocabulary is built: count adjacent symbol pairs across the corpus, pick the most frequent (ties break lexicographically), and merge it everywhere, left to right, without overlaps. Run it a few thousand times and you have byte-pair encoding. Solve it in Python or TypeScript.
Beam Search Decoder
Greedy decoding takes the best next token and never looks back — straight into garden paths. Keep the k best partial sequences alive at every step, with a product score and a clean tie-break, and watch k=2 escape a trap k=1 falls into. Solve it in Python or TypeScript.
Semantic Chunker (Token Budget)
Before you embed a document for retrieval, split it into chunks that fit a token budget — without slicing a sentence. The greedy packer fills each chunk with whole sentences until the next would overflow. Solve it in Python or TypeScript, with hidden tests.
Speculative Decoding: Accept Step
Speculative decoding speeds up LLM inference: a small draft model proposes tokens, the big target model verifies them in one pass. The accept/reject rule guarantees the output matches the target’s own distribution. Implement it. Solve it in Python or TypeScript, with hidden tests.
KV-Cache Eviction (Attention Sinks)
An LLM’s KV cache grows every token, so long chats must drop old entries without wrecking quality. StreamingLLM keeps the first few "attention sink" tokens plus a sliding window, evicting the middle. Compute the survivors. Solve it in Python or TypeScript, with hidden tests.
Constrained Decoding (Logit Masking)
How do you force an LLM to emit only valid JSON or a token your grammar allows? Mask the logits: set every disallowed token to −∞, then take the argmax over what remains. The backbone of structured output. Solve it in Python or TypeScript, with hidden tests.
Labs 12
The Tokenizer
A language model can't read words — it reads tokens. Watch your text shatter into sub-word chunks, race character vs word vs sub-word tokenizers head-to-head, then build Byte-Pair Encoding by hand: merge the most frequent pair over and over to grow a vocabulary and shrink the sequence. Five acts — shatter it, race three tokenizers, build BPE, tune the vocab size, and feel why 'strawberry' trips up an LLM.
The Dice Loader: Sampling
Don't read about temperature, top-k, and top-p — drive them. A language model turns logits into a probability over the next token, then rolls; three dials load those dice. Slide temperature to sharpen or flatten the odds, clamp the tail with top-k or top-p, and watch the bar chart renormalize live — then hit Sample and roll. See exactly why an LLM sounds robotic at low temperature and unhinged at high.
The Meaning Map: Embeddings
Don't read about embeddings — explore the map. Every word becomes a point in space, placed so that closeness means similar meaning. Click a word to rank its nearest neighbours by cosine similarity, then run the famous vector arithmetic — king − man + woman lands right on queen, paris − france + japan lands on tokyo. Word embeddings and cosine similarity, made playable, with theory and a quiz.
The Spotlight: Attention
Don't read about attention — shine the spotlight yourself. In a Transformer, every word looks at every other word and weights how much to listen to each. Click a word to see where its attention goes — the verb leaning on its subject, the determiner pointing at its noun — as glowing links and a heatmap whose weights sum to one. Self-attention (the softmax(QKᵀ) heart of every LLM), made playable, with theory and a quiz.
The Memory Tax: KV-Cache Visualizer
Don't read about the KV cache — fill it. A model generating text pays a hidden rent: for every token, in every layer, it stores a key and a value so it never recomputes the past. Watch the cache stack up as it decodes, collapse it with grouped-query attention, then slam into the VRAM wall that decides how many users a GPU can serve — and quantize the cache to claw the memory back. Four acts — fill it, collapse it with GQA, hit the wall, quantize it.
The Understudy: Speculative Decoding
Don't read about speculative decoding — run it. A giant model reads one token at a time, slow, because each token drags all its weights through the GPU. So a small draft model guesses the next few tokens, and the big model verifies them all in one pass, keeping every guess it agrees with. Watch tokens get accepted and rejected, then tune the acceptance rate and draft length to see where 2-3x faster generation — with identical output — comes from. Three acts — feel the bottleneck, draft and verify, dial acceptance to speed.
The Rounding Room: Quantization
Don't read about quantization — round some weights yourself. A model is billions of 16-bit numbers, but do you need all 16? Snap each weight to a coarser grid — 8 bits, 4, even 2 — and watch the memory collapse while precision blurs. Discover why weights bunch near zero, why NF4 places its levels at the quantiles of a bell curve to beat plain INT4, and where quality finally breaks. Three acts — round one weight, quantize the whole bell curve, trade memory against quality at scale.
The Switchboard: Mixture of Experts
Don't read about mixture-of-experts — route the tokens yourself. How does a trillion-parameter model run like a small one? A router sends each token to just a few of many experts, so the model stores a giant and pays for a sliver. Watch tokens light up their chosen experts, slide expert count and top-k to pull total and active parameters apart, then send a burst with load-balancing off to watch one expert overload — and on to fix it. Three acts — route a token, store big pay small, balance the load.
The Assembly Line: Batching & Throughput
Don't read about batching — run the GPU. One request wastes a chip built for hundreds, because generating a token streams every weight through the GPU whether you use one sequence or fifty. Slide the batch up to watch throughput soar and cost-per-token fall, trade it against a latency budget, then race static batching (stalls on the slowest request) against continuous batching (never idles). Three acts — fill the GPU, throughput vs latency, static vs continuous.
Chain-of-Thought vs Direct
Don't read about chain-of-thought — watch it rescue a wrong answer. A model produces one token at a time, so asking it to blurt a final answer to a multi-step problem crams the whole computation into one step — and it often takes a tempting shortcut and gets it wrong. Asking it to think step by step lets it spend intermediate tokens working the problem, each step small enough to get right. Run the same problems both ways and see the shortcut fail and the chain succeed — made playable, with theory and a quiz.
All At Once — Text Diffusion Decoding
Don't read about diffusion LLMs — run one. An autoregressive model writes one token per pass; a text-diffusion model starts from a fully masked block and refines the whole thing in parallel over K denoising steps, emitting B/K tokens per pass. Watch a block go from ██████ to clean text, then tune the block size and step count to feel the speed-versus-coherence dial — and see why text diffusion (masking, discrete) is not image diffusion (Gaussian noise, continuous). Interactive, with theory and a quiz.
The Million-Token Bill — Sparse Attention
Don't read about long-context attention — run the numbers. Dense attention caches a key/value per token and scores every query against all of them, so at a million tokens memory grows linearly and compute grows with the square. Pull the two levers of compressed sparse attention — compress each token's K/V to a latent, and select only the top-k blocks per query — and watch the KV cache and attention FLOPs collapse, with the two savings multiplying (as in DeepSeek-V4). Drag the context to a million and see the ratios fall. Interactive, with theory and a quiz.
Interactive Tools 18
RAG Chunking Playground
Drop in any text and compare chunking strategies — fixed-size, recursive, by-sentence, by-paragraph — with overlap highlighted and an estimated token count per chunk. Stop guessing your chunk size; see exactly how your RAG pipeline will split a document.
Context Budget & Cost Planner
Add a system prompt, tool definitions, conversation history and retrieved context, then see your context window fill up and the cost per call — plus the bill at 1k and 1M requests — across model price tiers. An architecture planner, not a toy token counter.
LLM-as-Judge Rubric Builder
Define your evaluation criteria and a scoring scale, then generate a clean, copy-pasteable LLM-as-judge prompt you can drop into your eval pipeline — with the common pitfalls (position bias, verbosity bias, ties) called out. Turns eval theory into a prompt you can ship.
Tool-Schema Designer
Compose a tool/function definition field by field — name, description, parameters, required flags — and export valid tool-use JSON for the Claude and OpenAI formats, with the JSON Schema generated for you. Stop hand-writing function-calling schemas and fighting silent validation errors.
LLM Pricing Comparator
Set input and output tokens per call and your monthly request volume, and instantly compare cost per call and cost per month across GPT-4o, Claude, Gemini and open models — sorted cheapest-first with the spread shown. See why output tokens dominate and how a 10× price gap between models turns into real dollars.
Context Window Visualizer
Set the tokens for your system prompt, tools, chat history, RAG context and output reserve, pick a model window (8K–1M), and watch a stacked bar fill it. The moment it overflows, the tool shows exactly how much compaction would have to trim from history and retrieved context to make the request fit.
Token Counter
Paste any prompt, document or code and get an instant token estimate, the cost of sending it as input across popular models (Claude, GPT-4o, Gemini, Llama), and how much of each context window — 8K to 1M — it consumes. Tuned heuristics for prose vs code, no signup, nothing leaves your browser.
GPU VRAM Calculator
Pick a model (Llama, Mistral, Qwen, Phi, Mixtral or custom architecture), a precision (FP16 / INT8 / INT4), a context length and batch size — and get the real VRAM requirement: weights, KV cache (GQA-aware), runtime overhead, and a fits-or-not verdict across common GPUs from a T4 to an H200.
Fine-Tuning Cost Estimator
Pick a method (QLoRA / LoRA / full AdamW), model size, dataset and GPU, and get the napkin answer: training memory, how many cards you need to fit, GPU-hours from the 6·P·tokens FLOPs rule, wall-clock time and the rental bill — with the QLoRA-vs-full memory gap made painfully visible.
KV Cache Calculator
Pick a model architecture (GQA-aware presets from published configs), a cache dtype (FP16/FP8/INT4) and a context length, and see the KV cache laid bare: bytes per token, gigabytes per sequence, the whole batch — and how many full-context users actually fit beside the weights on a 24/48/80/141 GB card.
Latency Budget Builder
Assemble a chat request stage by stage — network, gateway, retrieval, rerank, prefill, decode — and watch the waterfall: time to first token, full response time, a stacked bar of where the milliseconds went, and which stage to attack first when the budget blows.
Model Comparison Table
Claude, GPT, Gemini, DeepSeek, Kimi and Llama side by side: context windows, list prices, open-vs-closed, and a blended per-token cost computed at your real input:output ratio — because a RAG app and a story generator should not read the same pricing table.
Training Compute Calculator
Put in a model size and a token budget and see the training compute it implies — the classic 6ND FLOPs, the GPU-hours and wall-clock time on your cluster, the dollar cost, and how your token budget compares to the Chinchilla compute-optimal 20× rule. Every number is editable, from GPU peak FLOP/s to model-FLOPs-utilization.
Prompt Caching Savings Calculator
Prompt caching is the biggest cost lever on repetitive LLM workloads — a shared system prompt or RAG context read on every call is charged at a fraction of the price once cached. Put in your prompt shape (cached prefix vs fresh tokens), a cache hit rate, and the provider's cache-write and cache-read multipliers, and see your effective cost per request and the percentage saved versus paying full price every time.
Agent Loop Cost Estimator
An AI agent cost estimator. Enter the steps per task, the tokens added each step, and your model’s input/output prices to see the cost per task, per day and per month — and why re-sending a growing context each step makes cost scale with the square of the steps, not linearly. Shows where prompt caching helps.
GPU Rental Price Reference
A cloud GPU price reference and cost estimator. Compare approximate on-demand hourly rates for common GPUs (T4, L4, A10G, A100, H100) and estimate what a training or inference job costs from the GPU count and hours. Rates are approximate and drift over time — always confirm current pricing with your provider.
NPU Model-Fit Calculator
An on-device AI planner keyed to the NPU, not the GPU. Enter your device’s TOPS (e.g. a 75-TOPS Snapdragon X2) and unified RAM, pick a model size and quantization, and see two things the VRAM-fit calculators miss: whether the weights fit in memory, and how fast the NPU can prefill a prompt — the compute-bound step that TOPS actually governs. Get an escalate-to-cloud verdict when a prompt is too long to stay responsive locally.
EU AI Act Timeline
An engineering-oriented timeline of the EU AI Act: the statutory milestones (entry into force, prohibitions, GPAI obligations, high-risk rules) and what each one asks a builder to produce — model cards, logging, risk documentation, human-oversight design. Filter by your system type to see which obligations land and when. This is a plain-language planning aid for engineers, not legal advice, and it flags where proposed changes (the Digital Omnibus) may move dates that are not yet settled.
About LLM engineering
Building with large language models is a distinct discipline from training them. It's the engineering around the model: writing prompts that hold up in production, controlling how tokens are decoded, budgeting the context window, serving inference at scale, and keeping cost and latency in check as traffic grows.
These pieces connect the interface every model shares (the prompt and its tokens) down to the softmax that turns logits into the next-token probabilities. Whether you're shipping a chatbot, an agent or a RAG app, this is the layer where reliability, speed and spend are actually won — the difference between a demo and a product.