Design the systems
that run the models.
Build real AI architectures step by step — inference serving, context windows, RAG retrieval, agent loops and guardrails — through interactive diagrams that grow one concept at a time. The LLM-era counterpart to classic system design.
The ideas behind every AI system
Every design below recombines the same LLM-era building blocks — tokens and context windows, retrieval, tool-calling and evals. Get oriented with our handbooks and AI interactive tools, then spot the patterns in each design.
RAG Chunking Playground
Tune chunk size & overlap and see retrieval quality move.
01Context & Cost Planner
Budget tokens across prompt, history and output.
02Tool Schema Designer
Shape the JSON schemas an agent calls tools with.
03LLM Judge Builder
Build an eval rubric to score non-deterministic output.
Browse all AI designs
61 designsDesign a Conversational AI
Build a production conversational AI system (think ChatGPT). See how the request path splits an inference gateway from the model servers, how the context window is assembled and token-budgeted, how conversation memory is stored and recalled, how tokens stream back over a persistent connection, and how guardrails gate every prompt and response.
Design a RAG Pipeline
Build a retrieval-augmented generation pipeline. See how documents are chunked and embedded, how a vector store answers semantic search, how two-stage retrieval with reranking finds the best passages, how the prompt is grounded to stop hallucination, and how evals keep a quietly-drifting index honest.
Design an AI Agent System
Build an autonomous AI agent. See how the plan-act-observe loop turns a goal into action, how the model emits typed tool calls, how a sandboxed executor runs them safely, how working and long-term memory fit together, and how budgets and approval gates keep a multi-step agent from running away.
Design an LLM Inference Server
Build an LLM inference serving system. See how a request queue absorbs spiky traffic, how the prefill/decode split and continuous batching keep GPUs full, how the KV cache and paged attention make each token cheap, how tensor sharding fits a giant model, and how autoscaling rides demand — all balancing latency against throughput,.
Design a Recommendation System
Build a large-scale recommendation system. See how a two-stage retrieve-and-rank funnel picks the best few from millions, how two-tower embeddings and ANN generate candidates fast, how a heavy ranking model scores engagement, how a feature store stays consistent between training and serving, and how the feedback loop keeps recommendations fresh.
Design a Vector Database
Build a vector database. See why "k nearest of a billion vectors" needs its own index, how a distance metric ranks similarity, how IVF cells and an HNSW graph make search sub-linear, how product quantization fits billions in RAM, how metadata filtering and sharding hold up — and how ANN fails silently when you starve the search.
Design Semantic Search
Build a semantic search engine. See why keyword search misses meaning, how a single shared embedding model puts documents and queries in one space, how the chunk→embed→index ingest path is built, how hybrid BM25 + vector fusion catches exact terms, how a cross-encoder reranks the shortlist — and how a model-version upgrade silently randomizes results.
Design an LLM Gateway
Build an LLM gateway and model router. See why apps should call one provider-agnostic API instead of vendor SDKs, how adapters normalize every provider, how a capability-first router picks a model, how retries and failover survive a provider outage, how per-tenant limits and budgets isolate a shared quota, how caching cuts cost and latency — and how cost-only routing silently wrecks quality.
Design an LLM Cache
Build a prompt & response caching layer for LLMs. See why repeated and near-duplicate prompts should skip the model, how an exact-match cache keys on the normalized request, how a semantic cache reuses answers above a tuned similarity threshold, how prompt-prefix (KV) reuse cheapens even misses, how TTL and invalidation keep it fresh — and how a loose threshold silently serves wrong answers.
Design a Fine-Tuning Pipeline
Build an LLM fine-tuning and training pipeline. See how datasets are curated (the real work), how a pretrained base is adapted with LoRA/PEFT, how the distributed training loop is monitored, how a held-out eval gate decides promotion, how a versioned model registry enables rollback, how canary deploys ship safely, and how the production data flywheel compounds — plus why eval-set contamination inflates metrics silently — through an interactive diagram.
Design a Feature Store
Build a feature store for ML. See how one feature definition kills training/serving skew, how a shared pipeline computes features once, how the offline store serves point-in-time-correct training data, how the online store serves millisecond lookups, how a versioned registry makes features reusable assets, how batch + streaming keep them fresh, and how drift monitoring catches silent decay.
Design Content Moderation
Build a content-moderation pipeline (text + image). See how a staged funnel hash-matches known-bad content, how text and multimodal classifiers emit per-category scores, how OCR closes the text-in-image loophole, how a policy engine maps scores to graduated actions, how a human review queue handles the uncertain middle, how appeals and a retraining loop fight adversarial evasion — and why fully trusting the classifier fails — through an interactive diagram.
Design an AI Coding Assistant
Build an AI coding assistant (like Copilot or Cursor). See how inline completion / tab prediction meets a sub-second latency budget, how context is assembled with fill-in-the-middle, how repo-aware retrieval grounds completions in the codebase, how a code-specialized model streams suggestions, how debounce/cancel/cache win the milliseconds, how acceptance-rate telemetry measures quality, how an agentic chat mode handles multi-file edits, and why starving the context yields confident wrong code — through an interactive diagram.
Design Multi-Agent Orchestration
Build a multi-agent orchestration / workflow engine. See how an orchestrator decomposes a goal across specialist agents, when NOT to go multi-agent, how shared state coordinates them, how a durable workflow engine checkpoints and resumes, how typed handoffs stop telephone-game degradation, how fan-out/fan-in parallelizes independent work, how budgets and termination prevent runaways, and why uncapped agents loop and burn unbounded cost — through an interactive diagram.
Design LLM Eval & Observability
Build an LLM evaluation and observability pipeline. See how offline evals and online observability form two loops, how tracing every call is the foundation, how versioned golden datasets become your real benchmark, how programmatic/LLM-judge/human scorers combine, how a CI regression gate blocks regressions, how production monitoring catches drift, how failures feed back as fixtures — and why an unvalidated LLM judge silently corrupts every metric — through an interactive diagram.
Design an LLM Guardrails System
Build an LLM guardrails / safety-filter system. See how a safety pipeline wraps every call, how input guardrails screen PII, prompt-injection and scope, why injection needs defense-in-depth, how output guardrails check toxicity, PII, schema and groundedness, how violations are handled gracefully, how cheap-first layering keeps it affordable, why both over- and under-blocking fail — and how trusting untrusted input lets injection through — through an interactive diagram.
Design a Text-to-Image Service
Build a text-to-image generation service like Midjourney, DALL·E or hosted Stable Diffusion. See why synchronous generation fails, how the async job pattern accepts fast and works later, how a queue absorbs bursts against fixed GPU capacity, what the diffusion denoising loop actually does, how batching and autoscaling keep the GPU fleet cost-effective, how object storage + CDN deliver images, and why moderation is required on both the prompt and the image.
Design a Speech-to-Text Service
Build a speech-to-text / ASR service like Whisper-at-scale or Deepgram. See why batch and streaming transcription have opposite constraints, how a gateway splits the paths over one GPU fleet, how VAD and overlapping chunks feed a fixed model window, what the ASR inference pipeline does, how batching and autoscaling keep GPUs economical, and how post-processing (punctuation, timestamps, diarization) turns raw tokens into a usable transcript.
Design a Text-to-Speech Service
Build a text-to-speech / voice synthesis service like ElevenLabs or Play.ht. See why natural TTS is a pipeline (text normalization → acoustic model → vocoder), how batch and streaming differ, why time-to-first-audio is the metric for voice agents, how voices and cloning work, how caching common phrases skips the GPU, how GPU scaling protects the latency budget, and why cloning demands consent and watermarking.
Design a Realtime Voice Agent
Build a realtime voice AI agent like ChatGPT voice mode or an AI phone agent. See why a sequential STT→LLM→TTS pipeline feels robotic, how streaming and pipelining the three stages cut per-turn latency, how turn-taking (endpointing + barge-in) makes it feel human, how the interruptible orchestrator manages state and clean cancellation, where the latency budget goes, and how tools and transport (WebRTC/SIP) fit in.
Design an AI Answer Engine
Build an AI answer engine like Perplexity. See why a bare LLM is stale and hallucinates with no sources, how retrieve-then-generate (RAG over the live web) grounds answers, how a query planner rewrites and decomposes questions, how search + fetch + rerank build a tight evidence set, how grounded generation cites every claim, how citation verification catches unsupported statements, and how to defend against prompt injection from web content.
Design an Embeddings Service
Build an embeddings service that powers semantic search, RAG and recommendations. See why embedding needs a shared service, how online (low-latency query) and batch (millions of docs) modes differ, what the model outputs, how a batched GPU pipeline indexes a corpus, the critical rule that query and corpus must share the same model version (so a model change re-embeds everything), and how caching by (text, version) saves cost.
Design a Reranking Service
Build a reranking service — the precision tier of two-stage retrieval for search and RAG. See why vector (bi-encoder) search has great recall but rough ordering, how retrieve-wide-then-rerank-narrow works, why a cross-encoder scores query and candidate jointly for precision, how the top-K latency/cost tradeoff is tuned, how to serve it batched and cached, and why recall gates precision so both stages are essential.
Design a GraphRAG System
Build a GraphRAG system — knowledge-graph-augmented retrieval. See why vanilla vector RAG fails on multi-hop and global questions, how a knowledge graph of entities and relationships lets retrieval traverse connections, how LLM extraction builds the graph, how community detection and hierarchical summaries enable global questions, how local vs global query modes work, and why hybrid graph+vector retrieval is the strongest form.
Design a Multimodal RAG System
Build a multimodal RAG system that answers questions grounded in images, charts, tables and PDFs. See why text-only RAG loses visual information, how documents are parsed into multimodal chunks, unified multimodal embeddings vs convert-to-text, cross-modal retrieval, why generation needs a vision-language model that actually sees the retrieved images, how assets are stored and served, and the modality-gap and cost tradeoffs.
Design an Agent Memory System
Build a long-term memory system for an LLM agent or chatbot. See why stuffing the whole history into context fails, how short-term (context window) and long-term (persistent) memory differ, how the write path extracts salient facts, how context is assembled under a token budget, how memories live in vector + structured stores, how recall is RAG over memory, and why consolidation and forgetting keep memory coherent.
Design a Code Execution Sandbox
Build a secure code execution sandbox for untrusted, LLM-generated (or user-submitted) code — the runtime behind code agents, online judges and notebooks. See why running code directly is a full compromise, how containers and microVMs isolate it, how resource limits stop hangs and fork bombs, the submit→isolate→capture pipeline, why sandboxes must be ephemeral, how warm pools keep it fast, and how egress control prevents exfiltration.
Design an LLM Router
Build an LLM router that sends each request to the cheapest model that can handle it. See why one-model-for-all overpays on the easy majority and fails the hard minority, how a fast difficulty classifier provides the routing signal, how tiered routing with a cheap-first cascade survives classifier errors, how semantic routing to specialists is cheaper and better, how caching and multi-provider fallback add resilience, and why the quality bar comes before cost.
Design a Batch Inference System
Build an offline batch LLM inference system that grinds millions of prompts at maximum throughput per dollar. See why looping the online API fails, how the async job model works, how huge and continuous batching saturate the GPU, how a sharded restartable pipeline survives failures, how length bucketing and prefix caching squeeze each batch, and how checkpointing unlocks cheap spot GPUs and batch-discount pricing.
Design a Synthetic Data Pipeline
Build a synthetic data pipeline that uses LLMs to generate training and eval data. See why naive "generate 10,000 examples" gives low-diversity, error-prone data, how a strong teacher model generates candidates, how diversity steering avoids mode collapse, how quality filtering and verification (execute code, ground facts) keep only good examples, how the data powers distillation/augmentation/evals, and how to avoid model collapse and contamination.
Design an RLHF Pipeline
Build an RLHF alignment pipeline. See why a pretrained base model is capable but unaligned, how supervised fine-tuning teaches instruction-following, why preference comparisons beat demonstrations, how a reward model turns human comparisons into a dense score, how PPO optimizes the policy with a KL constraint to prevent reward hacking, the DPO and RLAIF alternatives, and the reward-hacking, overoptimization and labeler-quality traps.
Design a Document AI Pipeline
Build a document AI / intelligent document processing pipeline that turns PDFs, scans and forms into structured data. See why plain OCR gives characters but loses structure, how preprocessing and OCR produce located text, how layout analysis recovers 2D structure (tables, key-value pairs), how templates vs ML vs VLM extraction produce schema JSON, how validation and confidence catch misreads and hallucinated fields, and how human-in-the-loop review keeps it accurate.
Design ChatGPT
Design ChatGPT itself — a chat product serving hundreds of millions of weekly users and billions of prompts a day. See why the serving path must be stateless, how conversations shard by id, how the context is rebuilt every turn under a token budget, how a model router becomes the #1 cost lever, how prefix caching kills redundant prefill, how continuous batching squeezes the GPU fleet, and the load-shedding playbook for a capacity crunch.
Design a Computer-Use Agent
Build a computer-use / browser agent — the agent whose only API is the screen. See how the see-think-act loop works one verified action at a time, why perception mixes screenshots with accessibility trees, how set-of-marks grounding turns "click Submit" into coordinates, why execution lives in a disposable sandboxed VM, how every action is verified against the next frame, where approval gates catch irreversible clicks, and how task memory survives 60-step workflows — through an interactive diagram where you can change the page mid-task.
Design a Deep Research Agent
Build a deep research agent — the system behind every "deep research" product. See how a lead agent decomposes the question into a coverage plan, why parallel sub-researchers get isolated contexts, how the search-read-note loop compounds understanding, why evidence lives as claims-paired-with-sources, how gap-driven iteration ends under a hard budget, and how citation verification keeps the final report honest — through an interactive diagram where you can poison a source.
Design an AI Meeting Notetaker
Build an AI meeting notetaker. See how a capture bot joins calls with per-speaker audio, why consent is architecture rather than a checkbox, how streaming ASR and diarization turn chaos into an attributed transcript, why extracted decisions and action items must cite timestamped segments, how meeting memory answers questions across months of calls, and how artifacts land in the tools where work happens — through an interactive diagram where you can unleash crosstalk.
Design a Text-to-Video System
Build a Sora-style text-to-video service. See why minutes-long renders force an async job spine, how safety gates both prompts and pixels, why an LLM expands prompts toward the training distribution, how a GPU scheduler runs queue economics with tiers and preemption, how diffusion transformers denoise spacetime latents jointly for coherent motion, why C2PA provenance and watermarking ship on every render, and how draft-then-final rendering makes iteration affordable — through an interactive diagram with a viral-surge chaos mode.
Design an Adaptive AI Tutor
Build an adaptive tutoring system like Khanmigo or Duolingo's AI tutor. See why a fixed question order fails every learner, how a per-skill mastery model picks the next problem, why grading routes structured answers to a deterministic checker and reserves the LLM for open-ended judgment, how the mastery feedback loop closes, why explanations are grounded in a curriculum knowledge base instead of freely generated, how a Socratic hint ladder avoids just giving away the answer, and how a long-term learner profile drives spaced repetition.
Design an AI Code Review Bot
Build an automated PR review bot like CodeRabbit or Graphite. See why cheap deterministic linters run before any LLM call, how review is scoped to the diff plus just enough context, how a repo embedding index (RAG over the codebase) supplies cross-file context the diff alone can't show, why every finding is confidence-gated before posting, how a comment ledger prevents re-pushes from spamming old feedback, how secrets are redacted before they ever reach the LLM, and how developer reactions tune down false positives over time.
Design an Agent Control Plane
Build the platform layer that runs a whole fleet of production AI agents — not how one agent coordinates a task, but how an organization deploys, versions, budgets, permissions, monitors, and can instantly kill any of potentially hundreds of independently-running agents. See why ad-hoc agent scripts are an operational blind spot, a central agent registry with versioning, server-enforced per-agent tool permissions, hard cost budgets checked per step, an out-of-band kill switch, full execution tracing, and canary rollout of new agent versions.
Design a Model Registry (MLOps)
Build the platform that versions, evaluates, promotes and can instantly roll back trained ML models — the governance layer between "a model finished training" and "a model is safely serving production traffic." See why an artifact alone is meaningless without lineage, staged promotion through gated environments, automated evaluation gates that fail closed, shadow/canary deployment because offline metrics aren't sufficient, instant rollback via immutable versioning, and production drift monitoring.
Design a GPU Cluster Scheduler
Build a scheduler for a shared pool of expensive GPUs across many teams. See why naive FIFO scheduling starves large distributed jobs and fragments capacity, gang scheduling for all-or-nothing distributed training, topology-aware placement, priority and preemption for latency-sensitive inference versus long-running batch training, fair-share quotas across competing teams, checkpoint-and-resume, and reconciling the scheduler's bookkeeping against real cluster state.
Design a Data Labeling Platform
Build a human annotation platform like Scale AI or Labelbox — sourcing labels from people, not generating them programmatically. See why a single annotator's answer can't be trusted as ground truth, multi-annotator consensus, secretly-mixed gold-standard items that measure labeler accuracy in real time, skill-based task routing, active learning to prioritize which unlabeled examples matter most, and aligning labeler pay with measured quality.
Design a Prompt Management & A/B Testing Platform
Build the platform that versions, evaluates, and safely A/B tests prompts in a production LLM application — a much faster-moving artifact than model weights, often edited by non-engineers. See why hardcoded prompts force a full deploy for every tweak, externalizing prompts as versioned config, an offline eval gate before rollout, live A/B testing, guardrail metrics that must not regress, routine one-click rollback, and a review gate suited to non-engineer editors.
Design Realtime Speech Translation
Build a live speech-to-speech translator, chaining streaming ASR, MT, and TTS under a tight latency budget. See why record-then-translate feels like a walkie-talkie, streaming partial transcripts, why streaming translation must revise its own output as more context arrives, chunking policy, voice/prosody preservation, disfluency filtering, per-stage latency budgeting, and code-switching.
Design an AI Ad Creative Platform
Build a platform that generates ad creative at scale for advertisers, built on top of generative models rather than serving them directly. See why one hand-designed creative can't be tested at scale, programmatic variant generation, a brand-safety and ad-policy compliance gate that fails closed, rights and licensing tracking, dynamic multi-armed-bandit budget allocation, a performance feedback loop, and tiered generation cost control.
Design a Multi-Tenant AI SaaS Platform
Build an AI product serving many separate customer organizations from shared infrastructure. See why unscoped shared retrieval risks cross-tenant data leaks, enforcing isolation at the data layer, per-tenant customization via configuration, noisy-neighbor prevention with per-tenant quotas, per-tenant cost attribution, data-residency routing, automated onboarding, and defense-in-depth isolation that survives a bug in any single layer.
Design an Autonomous Email Agent
Build an AI agent that triages, drafts, and sends email on a user's behalf, where sending is often irreversible. See why auto-sending everything is dangerous, a human-approval gate scoped to action reversibility, priority triage, context assembly from thread history, tone/style matching, defending against indirect prompt injection hidden in received email, and a full audit trail.
Design a Sales CRM Agent
Build an AI agent embedded in a CRM that enriches leads, scores priority, drafts outreach, and suggests next actions — writing into a shared system of record and staying a suggester, not an autonomous actor, for relationship-sensitive interactions. See multi-source enrichment with provenance tracking, lead scoring, write-back validation that fails closed, next-best-action suggestion, personalization at scale, activity logging, and stale-data safeguards.
Design a Support Resolution Agent
Build an AI agent that triages, resolves, and escalates customer support tickets end-to-end. See why answering from general knowledge alone is dangerous when the real answer depends on account-specific state, grounding responses in real account/order data, confidence-based escalation instead of guessing, tiered auto/assisted/escalate resolution, seamless handoff context, outcome-based quality tracking, tone-aware de-escalation, and identity verification against social engineering.
Design a CI Test-Generation Agent
Build an agent that automatically generates tests for new and changed code in a CI pipeline. See why manual tests leave coverage gaps, coverage-gap analysis, generating tests against the real code, mutation testing to verify a test is actually meaningful, flaky-test detection before it poisons CI signal, why a human review gate is still required, and running expensive validation in a background lane.
Design a Personal Knowledge Assistant
Build an AI assistant grounded in a user's own private documents and notes, where privacy is the central architectural constraint. See why dumping years of personal content into one context window fails, retrieval over messy heterogeneous formats, continuous incremental ingestion, disambiguating with personal context, source attribution, encryption and minimizing third-party exposure, genuine deletion, and recency-aware retrieval for facts that change over time.
Design an Edge Inference Fleet
Build the platform that deploys and manages ML inference across a large fleet of edge devices — phones, IoT sensors, kiosks, cars. See why cloud-only inference fails at fleet scale, model compression for severe resource budgets, offline-capable on-device inference, staged rollout that's harder than server rollback, bandwidth-conscious delta updates, telemetry from devices you don't control, graceful degradation, and devices that self-heal their model version on reconnect.
Design an AI Video Dubbing System
Build a system that translates and re-synthesizes speech in existing video, synchronized to lip movements — an offline batch problem with its own timing and ethical constraints. See why naive translate-and-overlay drifts out of sync, timing-constrained translation, higher-quality offline voice cloning, the audio-vs-video lip-sync trade-off, cross-segment consistency, consent for voice likeness, review gates scaled by content stakes, and background audio preservation.
Deploy an LLM in a Customer’s Environment
Deploy a large language model inside a customer’s own environment — the constraint a forward deployed engineer meets when a regulated customer won’t send data to a public API. See why a hosted endpoint is a non-starter, right-sizing the model and precision to the fixed hardware they actually own, serving efficiently on limited GPUs, running with no network egress (air-gapped, no phone-home), shipping model and security updates as signed bundles into a locked-down environment, and getting observability out without exfiltrating customer data.
Design Secure Document Ingestion + RAG
Build a retrieval-augmented assistant over a customer’s sensitive internal documents, where security — not relevance — is the hard constraint. See why naive RAG leaks documents across users, classifying and tagging access on ingest, permission-aware retrieval that intersects relevance with what the user may see, redacting PII and secrets before the prompt and logs, defending against prompt injection carried inside untrusted documents, and citing sources plus auditing every access so the pipeline can pass a security review.
Design an Agent Payment Gateway
Build the platform that lets autonomous agents pay for things mid-task without draining an account — the agent rails, not the human card rails. See why a direct payment credential gives an agent unbounded, unscoped spending power, routing every payment through one mediated gateway, verifying a signed spending mandate (signature, scope, expiry) instead of trusting the agent's claim, enforcing per-transaction and cumulative spend caps BEFORE any settlement, machine-native x402 (HTTP 402) settlement with no human checkout, an immutable audit trail of every decision, and why the gateway must fail closed — so a runaway or hijacked agent is capped at the mandate, not the credit limit.
Design an MCP Security Gateway
Build the security layer between an AI host and the MCP servers it connects to — the MCP-protocol-specific threat surface, not a generic LLM guardrail. See why a direct connection implicitly trusts untrusted third-party servers, routing all MCP traffic through a mediating gateway, an explicit server/tool allowlist (deny by default), scanning tool descriptions for injection (attacker-controlled text the model reads as instructions — a channel unique to MCP), issuing narrowly-scoped short-lived tokens per server so a compromised one is bounded to its slice, egress control against exfiltration through tool results, an immutable audit trail, and why the gateway must fail closed.
Design an Agentic Browser Security Gateway
Build the enterprise perimeter that contains a whole fleet of agentic browsers without banning them — the fleet-level layer, not one agent's own injection defenses. See why ungoverned agentic browsers are an invisible exfiltration surface (and why a ban just creates shadow usage), routing every action through one gateway, a central org-wide policy engine, taint tracking so data from an untrusted web page can't drive a sensitive action (provenance beats detection), an action allowlist with risk classification, human-approval breakpoints for the irreversible minority, and a fleet-wide immutable audit trail.
Design a Hybrid Edge-Cloud Agent
Build an on-device agent that runs on the NPU by default and escalates hard queries to the cloud — but where privacy, not just confidence, decides what may leave the device. See why default-to-cloud is default-to-leak, answering on-device by default, a privacy classifier that gates escalation FIRST (must-stay-local data is answered locally even when the model is unsure), confidence-based escalation only for privacy-cleared queries (the cascade math lives in the LLM Router design), redaction to minimize what leaves, and an on-device escalation audit — capability and privacy reconciled by making the boundary a hard constraint confidence can never override.
Design an RL Environment Farm
Build the infrastructure that trains reasoning models — the environment-and-verifier side of RL with verifiable rewards, not the preference-tuning of RLHF. See why running rollouts inline in the trainer is unsafe and unscalable, sandboxed rollout workers that safely execute untrusted model-generated code, a scheduler that fans rollouts across a fleet and distributes the policy, a SOUND verifier (a gameable one makes the model learn the exploit — reward hacking), a task bank that serves the productive difficulty band (a curriculum, because learning needs reward variance), and a trajectory store that closes the loop — with reward-hacking and sandbox-escape chaos.
AI system design — frequently asked questions
What is AI system design?
AI system design is the practice of architecting production systems built around large language models and other AI — deciding how to serve inference, assemble and budget the context window, retrieve knowledge (RAG), store conversation memory, orchestrate tool-calling agents, and add guardrails and evals. It shares the rigor of classic system design but foregrounds LLM-specific concerns like tokens, latency-vs-cost, and non-determinism.
How is this different from classic system design?
Classic system design is about distributing data and traffic — caching, sharding, fan-out, consistency. AI system design adds a new axis: token budgets and context windows, vector retrieval quality, streaming token delivery, GPU inference serving, prompt and output guardrails, and evaluating non-deterministic model output. Many building blocks carry over (queues, caches, replicas); the trade-offs are new.
Are the AI system-design guides free?
Yes. Every guide is free, self-contained, and runs in your browser with no sign-up. You build each system step by step through an interactive diagram.
Which AI system design should I start with?
Start with the Conversational AI design. It establishes the core LLM serving path — inference gateway, context assembly and token budgeting, conversation memory, streaming, and guardrails — that every other AI system builds on.
Will these help with AI engineering interviews?
Yes. As teams ship LLM features, interviews increasingly probe how you would design a chat system, a RAG pipeline, or an agent — how you manage context, control cost and latency, retrieve reliably, and keep output safe. These guides foreground exactly those trade-offs.
Pick a system. Start building.
Every guide is interactive and self-contained — no setup, no sign-up. Start with the Conversational AI build, or explore the classic system-design library.