AI system design interview practice

61 real AI architectures — inference serving, RAG, agents, guardrails, evals — each built one decision at a time in a diagram you drive. Free, no sign-up.

61 builds572 guided stepsClassic system design

New here? Watch first · 44 minThe basics of AI system designOne support chatbot, built piece by piece. No background needed.

Start hereConversational AI The core LLM serving path every other design builds on · 9 steps Start building

61 designs

01Start hereLLM

Conversational AI

Build a production conversational AI system (think ChatGPT). See how the request path splits an inference gateway from the model servers, how the context window is assembled and token-budgeted, how conversation memory is stored and recalled, how tokens stream back over a persistent connection, and how guardrails gate every prompt and response.

You build Inference Gateway · Context Builder · Vector / KB +6

Intermediate 9 steps 02RAG

RAG Pipeline

Build a retrieval-augmented generation pipeline. See how documents are chunked and embedded, how a vector store answers semantic search, how two-stage retrieval with reranking finds the best passages, how the prompt is grounded to stop hallucination, and how evals keep a quietly-drifting index honest.

You build RAG Gateway · Vector Store · Document Store +6

Intermediate 9 steps 03Agents

AI Agent System

Build an autonomous AI agent. See how the plan-act-observe loop turns a goal into action, how the model emits typed tool calls, how a sandboxed executor runs them safely, how working and long-term memory fit together, and how budgets and approval gates keep a multi-step agent from running away.

You build Agent Loop · LLM (Planner) · Tool Executor +5

Advanced 9 steps 04Inference

LLM Inference Server

Build an LLM inference serving system. See how a request queue absorbs spiky traffic, how the prefill/decode split and continuous batching keep GPUs full, how the KV cache and paged attention make each token cheap, how tensor sharding fits a giant model, and how autoscaling rides demand — all balancing latency against throughput,.

You build Router · Request Queue · GPU Workers +5

Advanced 9 steps 05Recommenders

Recommendation System

Build a large-scale recommendation system. See how a two-stage retrieve-and-rank funnel picks the best few from millions, how two-tower embeddings and ANN generate candidates fast, how a heavy ranking model scores engagement, how a feature store stays consistent between training and serving, and how the feedback loop keeps recommendations fresh.

You build Recs Gateway · Candidate Gen · Item Index +5

Advanced 9 steps 06Retrieval

Vector Database

Build a vector database. See why "k nearest of a billion vectors" needs its own index, how a distance metric ranks similarity, how IVF cells and an HNSW graph make search sub-linear, how product quantization fits billions in RAM, how metadata filtering and sharding hold up — and how ANN fails silently when you starve the search.

You build Query API · Vector Segments · Ingest / Upsert +5

Advanced 9 steps 07Retrieval

Semantic Search

Build a semantic search engine. See why keyword search misses meaning, how a single shared embedding model puts documents and queries in one space, how the chunk→embed→index ingest path is built, how hybrid BM25 + vector fusion catches exact terms, how a cross-encoder reranks the shortlist — and how a model-version upgrade silently randomizes results.

You build Search API · Embedding Model · Document Store +6

Intermediate 9 steps 08LLM

LLM Gateway

Build an LLM gateway and model router. See why apps should call one provider-agnostic API instead of vendor SDKs, how adapters normalize every provider, how a capability-first router picks a model, how retries and failover survive a provider outage, how per-tenant limits and budgets isolate a shared quota, how caching cuts cost and latency — and how cost-only routing silently wrecks quality.

You build LLM Gateway · Provider A · Provider B +6

Intermediate 9 steps 09LLM

LLM Cache

Build a prompt & response caching layer for LLMs. See why repeated and near-duplicate prompts should skip the model, how an exact-match cache keys on the normalized request, how a semantic cache reuses answers above a tuned similarity threshold, how prompt-prefix (KV) reuse cheapens even misses, how TTL and invalidation keep it fresh — and how a loose threshold silently serves wrong answers.

You build Cache API · LLM Provider · Exact Cache +5

Intermediate 7 steps 10LLM

Fine-Tuning Pipeline

Build an LLM fine-tuning and training pipeline. See how datasets are curated (the real work), how a pretrained base is adapted with LoRA/PEFT, how the distributed training loop is monitored, how a held-out eval gate decides promotion, how a versioned model registry enables rollback, how canary deploys ship safely, and how the production data flywheel compounds — plus why eval-set contamination inflates metrics silently — through an interactive diagram.

You build Raw Data · Curate & Format · Training Set +6

Advanced 9 steps 11ML

Feature Store

Build a feature store for ML. See how one feature definition kills training/serving skew, how a shared pipeline computes features once, how the offline store serves point-in-time-correct training data, how the online store serves millisecond lookups, how a versioned registry makes features reusable assets, how batch + streaming keep them fresh, and how drift monitoring catches silent decay.

You build Raw Events · Feature Pipeline · Offline Store +6

Advanced 8 steps 12Safety

Content Moderation

Build a content-moderation pipeline (text + image). See how a staged funnel hash-matches known-bad content, how text and multimodal classifiers emit per-category scores, how OCR closes the text-in-image loophole, how a policy engine maps scores to graduated actions, how a human review queue handles the uncertain middle, how appeals and a retraining loop fight adversarial evasion — and why fully trusting the classifier fails — through an interactive diagram.

You build Moderation API · Case Store · Text Classifier +6

Advanced 9 steps 13LLM

AI Coding Assistant

Build an AI coding assistant (like Copilot or Cursor). See how inline completion / tab prediction meets a sub-second latency budget, how context is assembled with fill-in-the-middle, how repo-aware retrieval grounds completions in the codebase, how a code-specialized model streams suggestions, how debounce/cancel/cache win the milliseconds, how acceptance-rate telemetry measures quality, how an agentic chat mode handles multi-file edits, and why starving the context yields confident wrong code — through an interactive diagram.

You build Completion Svc · Context Builder · Repo Retrieval +6

Advanced 9 steps 14Agents

Multi-Agent Orchestration

Build a multi-agent orchestration / workflow engine. See how an orchestrator decomposes a goal across specialist agents, when NOT to go multi-agent, how shared state coordinates them, how a durable workflow engine checkpoints and resumes, how typed handoffs stop telephone-game degradation, how fan-out/fan-in parallelizes independent work, how budgets and termination prevent runaways, and why uncapped agents loop and burn unbounded cost — through an interactive diagram.

You build Orchestrator · Researcher · Analyst +5

Advanced 9 steps 15Evals

LLM Eval & Observability

Build an LLM evaluation and observability pipeline. See how offline evals and online observability form two loops, how tracing every call is the foundation, how versioned golden datasets become your real benchmark, how programmatic/LLM-judge/human scorers combine, how a CI regression gate blocks regressions, how production monitoring catches drift, how failures feed back as fixtures — and why an unvalidated LLM judge silently corrupts every metric — through an interactive diagram.

You build Trace Collector · Trace Store · Eval Set +6

Advanced 9 steps 16Safety

LLM Guardrails System

Build an LLM guardrails / safety-filter system. See how a safety pipeline wraps every call, how input guardrails screen PII, prompt-injection and scope, why injection needs defense-in-depth, how output guardrails check toxicity, PII, schema and groundedness, how violations are handled gracefully, how cheap-first layering keeps it affordable, why both over- and under-blocking fail — and how trusting untrusted input lets injection through — through an interactive diagram.

You build Guarded LLM API · LLM · Input Guardrails +6

Advanced 9 steps 17Diffusion

Text-to-Image Service

Build a text-to-image generation service like Midjourney, DALL·E or hosted Stable Diffusion. See why synchronous generation fails, how the async job pattern accepts fast and works later, how a queue absorbs bursts against fixed GPU capacity, what the diffusion denoising loop actually does, how batching and autoscaling keep the GPU fleet cost-effective, how object storage + CDN deliver images, and why moderation is required on both the prompt and the image.

You build API · Job Queue · GPU Worker +5

Advanced 10 steps 18Speech

Speech-to-Text Service

Build a speech-to-text / ASR service like Whisper-at-scale or Deepgram. See why batch and streaming transcription have opposite constraints, how a gateway splits the paths over one GPU fleet, how VAD and overlapping chunks feed a fixed model window, what the ASR inference pipeline does, how batching and autoscaling keep GPUs economical, and how post-processing (punctuation, timestamps, diarization) turns raw tokens into a usable transcript.

You build Gateway · Batch Queue · ASR Worker +4

Advanced 10 steps 19Speech

Text-to-Speech Service

Build a text-to-speech / voice synthesis service like ElevenLabs or Play.ht. See why natural TTS is a pipeline (text normalization → acoustic model → vocoder), how batch and streaming differ, why time-to-first-audio is the metric for voice agents, how voices and cloning work, how caching common phrases skips the GPU, how GPU scaling protects the latency budget, and why cloning demands consent and watermarking.

You build Gateway · Text Front-end · Acoustic Model +4

Advanced 10 steps 20Voice

Realtime Voice Agent

Build a realtime voice AI agent like ChatGPT voice mode or an AI phone agent. See why a sequential STT→LLM→TTS pipeline feels robotic, how streaming and pipelining the three stages cut per-turn latency, how turn-taking (endpointing + barge-in) makes it feel human, how the interruptible orchestrator manages state and clean cancellation, where the latency budget goes, and how tools and transport (WebRTC/SIP) fit in.

You build Transport · Orchestrator · Streaming STT +4

Advanced 10 steps 21RAG

AI Answer Engine

Build an AI answer engine like Perplexity. See why a bare LLM is stale and hallucinates with no sources, how retrieve-then-generate (RAG over the live web) grounds answers, how a query planner rewrites and decomposes questions, how search + fetch + rerank build a tight evidence set, how grounded generation cites every claim, how citation verification catches unsupported statements, and how to defend against prompt injection from web content.

You build Answer Service · Query Planner · Search +4

Advanced 10 steps 22Embeddings

Embeddings Service

Build an embeddings service that powers semantic search, RAG and recommendations. See why embedding needs a shared service, how online (low-latency query) and batch (millions of docs) modes differ, what the model outputs, how a batched GPU pipeline indexes a corpus, the critical rule that query and corpus must share the same model version (so a model change re-embeds everything), and how caching by (text, version) saves cost.

You build Embeddings API · Online Embedder · Embedding Model +4

Advanced 10 steps 23Retrieval

Reranking Service

Build a reranking service — the precision tier of two-stage retrieval for search and RAG. See why vector (bi-encoder) search has great recall but rough ordering, how retrieve-wide-then-rerank-narrow works, why a cross-encoder scores query and candidate jointly for precision, how the top-K latency/cost tradeoff is tuned, how to serve it batched and cached, and why recall gates precision so both stages are essential.

You build Retrieval API · Retriever · Index +4

Advanced 10 steps 24RAG

GraphRAG System

Build a GraphRAG system — knowledge-graph-augmented retrieval. See why vanilla vector RAG fails on multi-hop and global questions, how a knowledge graph of entities and relationships lets retrieval traverse connections, how LLM extraction builds the graph, how community detection and hierarchical summaries enable global questions, how local vs global query modes work, and why hybrid graph+vector retrieval is the strongest form.

You build GraphRAG Svc · Extractor · Knowledge Graph +4

Advanced 10 steps 25RAG

Multimodal RAG System

Build a multimodal RAG system that answers questions grounded in images, charts, tables and PDFs. See why text-only RAG loses visual information, how documents are parsed into multimodal chunks, unified multimodal embeddings vs convert-to-text, cross-modal retrieval, why generation needs a vision-language model that actually sees the retrieved images, how assets are stored and served, and the modality-gap and cost tradeoffs.

You build MM-RAG Svc · Doc Parser · MM Embedder +4

Advanced 10 steps 26Agents

Agent Memory System

Build a long-term memory system for an LLM agent or chatbot. See why stuffing the whole history into context fails, how short-term (context window) and long-term (persistent) memory differ, how the write path extracts salient facts, how context is assembled under a token budget, how memories live in vector + structured stores, how recall is RAG over memory, and why consolidation and forgetting keep memory coherent.

You build Agent · Memory Writer · Context Builder +4

Advanced 10 steps 27Agents

Code Execution Sandbox

Build a secure code execution sandbox for untrusted, LLM-generated (or user-submitted) code — the runtime behind code agents, online judges and notebooks. See why running code directly is a full compromise, how containers and microVMs isolate it, how resource limits stop hangs and fork bombs, the submit→isolate→capture pipeline, why sandboxes must be ephemeral, how warm pools keep it fast, and how egress control prevents exfiltration.

You build Exec API · Sandbox · Resource Limits +4

Advanced 10 steps 28LLM

LLM Router

Build an LLM router that sends each request to the cheapest model that can handle it. See why one-model-for-all overpays on the easy majority and fails the hard minority, how a fast difficulty classifier provides the routing signal, how tiered routing with a cheap-first cascade survives classifier errors, how semantic routing to specialists is cheaper and better, how caching and multi-provider fallback add resilience, and why the quality bar comes before cost.

You build Router · Classifier · Cheap Model +4

Advanced 10 steps 29LLM

Batch Inference System

Build an offline batch LLM inference system that grinds millions of prompts at maximum throughput per dollar. See why looping the online API fails, how the async job model works, how huge and continuous batching saturate the GPU, how a sharded restartable pipeline survives failures, how length bucketing and prefix caching squeeze each batch, and how checkpointing unlocks cheap spot GPUs and batch-discount pricing.

You build Batch API · Work Queue · Scheduler +4

Advanced 10 steps 30LLM

Synthetic Data Pipeline

Build a synthetic data pipeline that uses LLMs to generate training and eval data. See why naive "generate 10,000 examples" gives low-diversity, error-prone data, how a strong teacher model generates candidates, how diversity steering avoids mode collapse, how quality filtering and verification (execute code, ground facts) keep only good examples, how the data powers distillation/augmentation/evals, and how to avoid model collapse and contamination.

You build Pipeline · Teacher Model · Diversity +3

Advanced 10 steps 31LLM

RLHF Pipeline

Build an RLHF alignment pipeline. See why a pretrained base model is capable but unaligned, how supervised fine-tuning teaches instruction-following, why preference comparisons beat demonstrations, how a reward model turns human comparisons into a dense score, how PPO optimizes the policy with a KL constraint to prevent reward hacking, the DPO and RLAIF alternatives, and the reward-hacking, overoptimization and labeler-quality traps.

You build Base Model · SFT · Reward Model +3

Advanced 10 steps 32Multimodal

Document AI Pipeline

Build a document AI / intelligent document processing pipeline that turns PDFs, scans and forms into structured data. See why plain OCR gives characters but loses structure, how preprocessing and OCR produce located text, how layout analysis recovers 2D structure (tables, key-value pairs), how templates vs ML vs VLM extraction produce schema JSON, how validation and confidence catch misreads and hallucinated fields, and how human-in-the-loop review keeps it accurate.

You build Doc AI Service · Preprocess + OCR · Layout +3

Advanced 10 steps 33Start hereLLM

ChatGPT

Design ChatGPT itself — a chat product serving hundreds of millions of weekly users and billions of prompts a day. See why the serving path must be stateless, how conversations shard by id, how the context is rebuilt every turn under a token budget, how a model router becomes the #1 cost lever, how prefix caching kills redundant prefill, how continuous batching squeezes the GPU fleet, and the load-shedding playbook for a capacity crunch.

You build Edge / Gateway · Chat Service · Conversation Store +6

Advanced 9 steps 34Agents

Computer-Use Agent

Build a computer-use / browser agent — the agent whose only API is the screen. See how the see-think-act loop works one verified action at a time, why perception mixes screenshots with accessibility trees, how set-of-marks grounding turns "click Submit" into coordinates, why execution lives in a disposable sandboxed VM, how every action is verified against the next frame, where approval gates catch irreversible clicks, and how task memory survives 60-step workflows — through an interactive diagram where you can change the page mid-task.

You build Agent Loop · VLM Pilot · Grounding +6

Advanced 9 steps 35Agents

Deep Research Agent

Build a deep research agent — the system behind every "deep research" product. See how a lead agent decomposes the question into a coverage plan, why parallel sub-researchers get isolated contexts, how the search-read-note loop compounds understanding, why evidence lives as claims-paired-with-sources, how gap-driven iteration ends under a hard budget, and how citation verification keeps the final report honest — through an interactive diagram where you can poison a source.

You build Research Lead · Sub-Researchers · Search / Fetch +5

Advanced 9 steps 36Speech

AI Meeting Notetaker

Build an AI meeting notetaker. See how a capture bot joins calls with per-speaker audio, why consent is architecture rather than a checkbox, how streaming ASR and diarization turn chaos into an attributed transcript, why extracted decisions and action items must cite timestamped segments, how meeting memory answers questions across months of calls, and how artifacts land in the tools where work happens — through an interactive diagram where you can unleash crosstalk.

You build Capture Bot · Consent & Policy · Streaming ASR +6

Intermediate 9 steps 37Diffusion

Text-to-Video System

Build a Sora-style text-to-video service. See why minutes-long renders force an async job spine, how safety gates both prompts and pixels, why an LLM expands prompts toward the training distribution, how a GPU scheduler runs queue economics with tiers and preemption, how diffusion transformers denoise spacetime latents jointly for coherent motion, why C2PA provenance and watermarking ship on every render, and how draft-then-final rendering makes iteration affordable — through an interactive diagram with a viral-surge chaos mode.

You build API / Job Queue · Prompt Safety · Prompt Expander +6

Advanced 9 steps 38LLM

Adaptive AI Tutor

Build an adaptive tutoring system like Khanmigo or Duolingo's AI tutor. See why a fixed question order fails every learner, how a per-skill mastery model picks the next problem, why grading routes structured answers to a deterministic checker and reserves the LLM for open-ended judgment, how the mastery feedback loop closes, why explanations are grounded in a curriculum knowledge base instead of freely generated, how a Socratic hint ladder avoids just giving away the answer, and how a long-term learner profile drives spaced repetition.

You build Tutor API · Problem Selector · Mastery Model +4

Advanced 10 steps 39Agents

AI Code Review Bot

Build an automated PR review bot like CodeRabbit or Graphite. See why cheap deterministic linters run before any LLM call, how review is scoped to the diff plus just enough context, how a repo embedding index (RAG over the codebase) supplies cross-file context the diff alone can't show, why every finding is confidence-gated before posting, how a comment ledger prevents re-pushes from spamming old feedback, how secrets are redacted before they ever reach the LLM, and how developer reactions tune down false positives over time.

You build Webhook / API · Static Analyzers · Diff + Context Assembler +4

Advanced 10 steps 40Agents

Agent Control Plane

Build the platform layer that runs a whole fleet of production AI agents — not how one agent coordinates a task, but how an organization deploys, versions, budgets, permissions, monitors, and can instantly kill any of potentially hundreds of independently-running agents. See why ad-hoc agent scripts are an operational blind spot, a central agent registry with versioning, server-enforced per-agent tool permissions, hard cost budgets checked per step, an out-of-band kill switch, full execution tracing, and canary rollout of new agent versions.

You build Agent Runtime · Control Plane API · Agent Registry +4

Advanced 10 steps 41LLM

Model Registry (MLOps)

Build the platform that versions, evaluates, promotes and can instantly roll back trained ML models — the governance layer between "a model finished training" and "a model is safely serving production traffic." See why an artifact alone is meaningless without lineage, staged promotion through gated environments, automated evaluation gates that fail closed, shadow/canary deployment because offline metrics aren't sufficient, instant rollback via immutable versioning, and production drift monitoring.

You build Production Serving · Model Registry · Versioned Artifacts +3

Advanced 10 steps 42Inference

GPU Cluster Scheduler

Build a scheduler for a shared pool of expensive GPUs across many teams. See why naive FIFO scheduling starves large distributed jobs and fragments capacity, gang scheduling for all-or-nothing distributed training, topology-aware placement, priority and preemption for latency-sensitive inference versus long-running batch training, fair-share quotas across competing teams, checkpoint-and-resume, and reconciling the scheduler's bookkeeping against real cluster state.

You build Scheduler API · GPU Inventory · Pending Jobs +4

Advanced 10 steps 43Evals

Data Labeling Platform

Build a human annotation platform like Scale AI or Labelbox — sourcing labels from people, not generating them programmatically. See why a single annotator's answer can't be trusted as ground truth, multi-annotator consensus, secretly-mixed gold-standard items that measure labeler accuracy in real time, skill-based task routing, active learning to prioritize which unlabeled examples matter most, and aligning labeler pay with measured quality.

You build Labeling API · Task Queue · Consensus Engine +3

Intermediate 10 steps 44LLM

Prompt Management & A/B Testing Platform

Build the platform that versions, evaluates, and safely A/B tests prompts in a production LLM application — a much faster-moving artifact than model weights, often edited by non-engineers. See why hardcoded prompts force a full deploy for every tweak, externalizing prompts as versioned config, an offline eval gate before rollout, live A/B testing, guardrail metrics that must not regress, routine one-click rollback, and a review gate suited to non-engineer editors.

You build LLM Call · Prompt Registry · Offline Eval Gate +3

Advanced 10 steps 45Speech

Realtime Speech Translation

Build a live speech-to-speech translator, chaining streaming ASR, MT, and TTS under a tight latency budget. See why record-then-translate feels like a walkie-talkie, streaming partial transcripts, why streaming translation must revise its own output as more context arrives, chunking policy, voice/prosody preservation, disfluency filtering, per-stage latency budgeting, and code-switching.

You build Streaming ASR · Incremental MT · Streaming TTS +3

Advanced 10 steps 46Diffusion

AI Ad Creative Platform

Build a platform that generates ad creative at scale for advertisers, built on top of generative models rather than serving them directly. See why one hand-designed creative can't be tested at scale, programmatic variant generation, a brand-safety and ad-policy compliance gate that fails closed, rights and licensing tracking, dynamic multi-armed-bandit budget allocation, a performance feedback loop, and tiered generation cost control.

You build Creative API · Variant Generator · Brand-Safety Gate +4

Advanced 10 steps 47LLM

Multi-Tenant AI SaaS Platform

Build an AI product serving many separate customer organizations from shared infrastructure. See why unscoped shared retrieval risks cross-tenant data leaks, enforcing isolation at the data layer, per-tenant customization via configuration, noisy-neighbor prevention with per-tenant quotas, per-tenant cost attribution, data-residency routing, automated onboarding, and defense-in-depth isolation that survives a bug in any single layer.

You build Tenant-Aware API · Tenant-Isolated Data · Per-Tenant Config +3

Advanced 10 steps 48Agents

Autonomous Email Agent

Build an AI agent that triages, drafts, and sends email on a user's behalf, where sending is often irreversible. See why auto-sending everything is dangerous, a human-approval gate scoped to action reversibility, priority triage, context assembly from thread history, tone/style matching, defending against indirect prompt injection hidden in received email, and a full audit trail.

You build Inbox · Draft Agent · Mail Sender +4

Advanced 10 steps 49Agents

Sales CRM Agent

Build an AI agent embedded in a CRM that enriches leads, scores priority, drafts outreach, and suggests next actions — writing into a shared system of record and staying a suggester, not an autonomous actor, for relationship-sensitive interactions. See multi-source enrichment with provenance tracking, lead scoring, write-back validation that fails closed, next-best-action suggestion, personalization at scale, activity logging, and stale-data safeguards.

You build CRM · Lead Enrichment · Source Provenance +4

Intermediate 10 steps 50Agents

Support Resolution Agent

Build an AI agent that triages, resolves, and escalates customer support tickets end-to-end. See why answering from general knowledge alone is dangerous when the real answer depends on account-specific state, grounding responses in real account/order data, confidence-based escalation instead of guessing, tiered auto/assisted/escalate resolution, seamless handoff context, outcome-based quality tracking, tone-aware de-escalation, and identity verification against social engineering.

You build Support API · Account/Order State · Triage + Confidence +3

Advanced 10 steps 51Agents

CI Test-Generation Agent

Build an agent that automatically generates tests for new and changed code in a CI pipeline. See why manual tests leave coverage gaps, coverage-gap analysis, generating tests against the real code, mutation testing to verify a test is actually meaningful, flaky-test detection before it poisons CI signal, why a human review gate is still required, and running expensive validation in a background lane.

You build CI Pipeline · Coverage-Gap Analyzer · Test Generator +4

Advanced 10 steps 52RAG

Personal Knowledge Assistant

Build an AI assistant grounded in a user's own private documents and notes, where privacy is the central architectural constraint. See why dumping years of personal content into one context window fails, retrieval over messy heterogeneous formats, continuous incremental ingestion, disambiguating with personal context, source attribution, encryption and minimizing third-party exposure, genuine deletion, and recency-aware retrieval for facts that change over time.

You build Continuous Ingestion · Answer + Citation · Personal Corpus Index +4

Advanced 10 steps 53Inference

Edge Inference Fleet

Build the platform that deploys and manages ML inference across a large fleet of edge devices — phones, IoT sensors, kiosks, cars. See why cloud-only inference fails at fleet scale, model compression for severe resource budgets, offline-capable on-device inference, staged rollout that's harder than server rollback, bandwidth-conscious delta updates, telemetry from devices you don't control, graceful degradation, and devices that self-heal their model version on reconnect.

You build Fleet Control Plane · Compression Pipeline · Compressed Model Versions +4

Advanced 10 steps 54Speech

AI Video Dubbing System

Build a system that translates and re-synthesizes speech in existing video, synchronized to lip movements — an offline batch problem with its own timing and ethical constraints. See why naive translate-and-overlay drifts out of sync, timing-constrained translation, higher-quality offline voice cloning, the audio-vs-video lip-sync trade-off, cross-segment consistency, consent for voice likeness, review gates scaled by content stakes, and background audio preservation.

You build Video Ingest · Transcribe + Diarize · Timing-Aware Translation +4

Advanced 10 steps 55Inference

Deploy an LLM in a Customer’s Environment

Deploy a large language model inside a customer’s own environment — the constraint a forward deployed engineer meets when a regulated customer won’t send data to a public API. See why a hosted endpoint is a non-starter, right-sizing the model and precision to the fixed hardware they actually own, serving efficiently on limited GPUs, running with no network egress (air-gapped, no phone-home), shipping model and security updates as signed bundles into a locked-down environment, and getting observability out without exfiltrating customer data.

You build Inference Gateway · Inference Server · Model Weights +5

Advanced 8 steps 56RAG

Secure Document Ingestion + RAG

Build a retrieval-augmented assistant over a customer’s sensitive internal documents, where security — not relevance — is the hard constraint. See why naive RAG leaks documents across users, classifying and tagging access on ingest, permission-aware retrieval that intersects relevance with what the user may see, redacting PII and secrets before the prompt and logs, defending against prompt injection carried inside untrusted documents, and citing sources plus auditing every access so the pipeline can pass a security review.

You build RAG Gateway · Ingest Pipeline · Retrieval +5

Advanced 8 steps 57Agents

Agent Payment Gateway

Build the platform that lets autonomous agents pay for things mid-task without draining an account — the agent rails, not the human card rails. See why a direct payment credential gives an agent unbounded, unscoped spending power, routing every payment through one mediated gateway, verifying a signed spending mandate (signature, scope, expiry) instead of trusting the agent's claim, enforcing per-transaction and cumulative spend caps BEFORE any settlement, machine-native x402 (HTTP 402) settlement with no human checkout, an immutable audit trail of every decision, and why the gateway must fail closed — so a runaway or hijacked agent is capped at the mandate, not the credit limit.

You build Merchant / API · Payment Gateway API · Mandate Verifier +3

Advanced 8 steps 58Agents

MCP Security Gateway

Build the security layer between an AI host and the MCP servers it connects to — the MCP-protocol-specific threat surface, not a generic LLM guardrail. See why a direct connection implicitly trusts untrusted third-party servers, routing all MCP traffic through a mediating gateway, an explicit server/tool allowlist (deny by default), scanning tool descriptions for injection (attacker-controlled text the model reads as instructions — a channel unique to MCP), issuing narrowly-scoped short-lived tokens per server so a compromised one is bounded to its slice, egress control against exfiltration through tool results, an immutable audit trail, and why the gateway must fail closed.

You build MCP Servers · MCP Gateway · Server Allowlist +3

Advanced 8 steps 59Agents

Agentic Browser Security Gateway

Build the enterprise perimeter that contains a whole fleet of agentic browsers without banning them — the fleet-level layer, not one agent's own injection defenses. See why ungoverned agentic browsers are an invisible exfiltration surface (and why a ban just creates shadow usage), routing every action through one gateway, a central org-wide policy engine, taint tracking so data from an untrusted web page can't drive a sensitive action (provenance beats detection), an action allowlist with risk classification, human-approval breakpoints for the irreversible minority, and a fleet-wide immutable audit trail.

You build Web / SaaS · Security Gateway · Policy Engine +3

Advanced 8 steps 60On-Device

Hybrid Edge-Cloud Agent

Build an on-device agent that runs on the NPU by default and escalates hard queries to the cloud — but where privacy, not just confidence, decides what may leave the device. See why default-to-cloud is default-to-leak, answering on-device by default, a privacy classifier that gates escalation FIRST (must-stay-local data is answered locally even when the model is unsure), confidence-based escalation only for privacy-cleared queries (the cascade math lives in the LLM Router design), redaction to minimize what leaves, and an on-device escalation audit — capability and privacy reconciled by making the boundary a hard constraint confidence can never override.

You build Cloud Model · On-Device Model · Privacy Classifier +3

Advanced 8 steps 61Reinforcement Learning

RL Environment Farm

Build the infrastructure that trains reasoning models — the environment-and-verifier side of RL with verifiable rewards, not the preference-tuning of RLHF. See why running rollouts inline in the trainer is unsafe and unscalable, sandboxed rollout workers that safely execute untrusted model-generated code, a scheduler that fans rollouts across a fleet and distributes the policy, a SOUND verifier (a gameable one makes the model learn the exploit — reward hacking), a task bank that serves the productive difficulty band (a curriculum, because learning needs reward variance), and a trajectory store that closes the loop — with reward-hacking and sandbox-escape chaos.

You build Rollout Worker · Rollout Scheduler · Verifier Service +2

Advanced 8 steps

Warm up on the building blocks

Every design above recombines the same LLM-era parts — tokens and context windows, retrieval, tool-calling and evals. These interactive tools let you turn each knob on its own first.

AI system design — frequently asked questions

What is AI system design?

AI system design is the practice of architecting production systems built around large language models and other AI — deciding how to serve inference, assemble and budget the context window, retrieve knowledge (RAG), store conversation memory, orchestrate tool-calling agents, and add guardrails and evals. It shares the rigor of classic system design but foregrounds LLM-specific concerns like tokens, latency-vs-cost, and non-determinism.

How is this different from classic system design?

Classic system design is about distributing data and traffic — caching, sharding, fan-out, consistency. AI system design adds a new axis: token budgets and context windows, vector retrieval quality, streaming token delivery, GPU inference serving, prompt and output guardrails, and evaluating non-deterministic model output. Many building blocks carry over (queues, caches, replicas); the trade-offs are new.

Are the AI system-design guides free?

Yes. Every guide is free, self-contained, and runs in your browser with no sign-up. You build each system step by step through an interactive diagram.

Which AI system design should I start with?

Start with the Conversational AI design. It establishes the core LLM serving path — inference gateway, context assembly and token budgeting, conversation memory, streaming, and guardrails — that every other AI system builds on.

Will these help with AI engineering interviews?

Yes. As teams ship LLM features, interviews increasingly probe how you would design a chat system, a RAG pipeline, or an agent — how you manage context, control cost and latency, retrieve reliably, and keep output safe. These guides foreground exactly those trade-offs.

Pick a system. Start building.

Every guide is interactive and self-contained — no setup, no sign-up. Start with the Conversational AI build, or explore the classic system-design library.

Start with Conversational AI Browse classic designs

All AI system designs