What is a conversational AI, really?

That sentence hides the hard parts. What exactly do we feed the model (and how much fits)? How does it remember earlier turns? How do we serve a giant model fast and affordably? And how do we keep the output safe?

A client, a gateway, a model

We put an Inference Gateway in the middle. The browser sends a turn; the gateway authenticates, rate-limits, and orchestrates everything behind it. This is the client → gateway → model backbone under every chat product.

Assemble the context window

A Context Builder composes the model input for every turn: the system prompt (who the assistant is), the conversation history, and the new message. The model is stateless, so the full context is rebuilt and resent on each request.

It won’t all fit — now what?

The builder works to a token budget. It pins the system prompt, keeps the most recent turns, and summarizes or drops older ones. For knowledge beyond the chat, it retrieves only the relevant passages from a vector store (RAG) instead of stuffing everything in.

Where does history live?

A Memory Service reads and writes the Conversation DB — the durable source of truth for every turn. Each request, the builder loads recent history from it; after each answer, the new turn is appended. The browser is just a fast cache of what the database already knows.

Inference is the expensive part

Generation runs on a dedicated Model Servers tier — a fleet of GPUs holding the model weights. Because each output token is a full pass through the model, this tier is the bottleneck we design everything else around: it gets its own queue, its own scaling, and its own failure handling.

Batching and the KV cache

The model servers use continuous batching — packing many in-flight chats into shared GPU passes — and a KV cache that stores attention state so each new token reuses prior work instead of recomputing the whole prompt. Every turn also emits usage events (tokens, latency) to the stream for billing and evals.

Stream the tokens back

The model generates tokens incrementally, and the gateway streams them straight to the browser over a persistent connection (server-sent events or WebSocket). The user sees the answer appear word by word. The metric that matters is time-to-first-token, not total time.

Guard the input and the output

A Guardrails layer screens both edges: the incoming prompt (block jailbreaks, abuse, PII leaks) before it reaches the model, and the outgoing answer before it streams to the user. Safety flags are emitted to the event stream so abuse can be detected and rate-limited over time.

Design a Conversational AI (ChatGPT-style) — A Guided System Design

System Design · step by stepDesign a Conversational AI

Step 1 / 9

Finished this one? 0 / 5 AI System Designs done

More AI System Designs

Design a RAG PipelineBuild a retrieval-augmented generation pipeline step by step. See how documents are chunked and embedded, how a vector store answers semantic search, how two-stage retrieval with reranking finds the best passages, how the prompt is grounded to stop hallucination, and how evals keep a quietly-drifting index honest — through an interactive diagram that grows with each concept.Read →
Design an AI Agent SystemBuild an autonomous AI agent step by step. See how the plan-act-observe loop turns a goal into action, how the model emits typed tool calls, how a sandboxed executor runs them safely, how working and long-term memory fit together, and how budgets and approval gates keep a multi-step agent from running away — through an interactive diagram that grows with each concept.Read →
Design an LLM Inference ServerBuild an LLM inference serving system step by step. See how a request queue absorbs spiky traffic, how the prefill/decode split and continuous batching keep GPUs full, how the KV cache and paged attention make each token cheap, how tensor sharding fits a giant model, and how autoscaling rides demand — all balancing latency against throughput, through an interactive diagram that grows with each concept.Read →
Design a Recommendation SystemBuild a large-scale recommendation system step by step. See how a two-stage retrieve-and-rank funnel picks the best few from millions, how two-tower embeddings and ANN generate candidates fast, how a heavy ranking model scores engagement, how a feature store stays consistent between training and serving, and how the feedback loop keeps recommendations fresh — through an interactive diagram that grows with each concept.Read →

Explore more from Vibe Engines

Handbooks →

Roadmaps →

Prompt Engineering Roadmap

System Designs →