The whole design, in writing
Learn AI system design by building an AI answer engine like Perplexity step by step. An interactive guide covering why asking an LLM alone hallucinates, the retrieve-then-generate (RAG-over-the-web) shape, query understanding and decomposition, live search + fetch + rerank, grounded generation with inline citations, citation verification, streaming answers with follow-ups, and the unhappy paths (hallucination despite sources, source trust, prompt injection, freshness, cost).
Every step of the build above, written out: the problem each piece solves, the option that was taken and the ones that were not, the numbers, and how it fails in production.
The big idea
What is an AI answer engine?
You ask a real question — "what changed in the latest React release?", "compare these two drugs' side effects". A plain chatbot will answer confidently from memory that's frozen at its training cutoff, sometimes right, sometimes hallucinated, and never with a source you can check. For questions where being current and correct matters, that's not good enough.
An AI answer engine (like Perplexity) answers by retrieving live sources first, then generating a grounded, cited answer from them — RAG over the open web. The pipeline: understand the question, search + fetch + rerank real sources, generate an answer using only those sources with inline citations, and verify the citations. Grounding and attribution are the whole point.
What the new pieces do
- Userclient
- Someone asking a real question that needs a current, trustworthy, cited answer — not a plausible-sounding guess.
Step 1 · Why not just ask the model?
The bare LLM fails
The simplest design: forward the question straight to an LLM and return its answer. Why is that inadequate for a question-answering product people should trust?
Just send the question to an LLM and return its answer. What's wrong?
Speed isn't the core issue. The problems are correctness and trust: the model's knowledge is frozen at its training cutoff (stale), it can hallucinate confidently, and it gives no sources to verify — unacceptable for a factual answer engine.
Fluency is exactly what it has — and that's the danger: it produces fluent, confident answers whether or not they're true, with no citations, and no knowledge of anything after its training cutoff.
A bare LLM can't know recent events, will sometimes fabricate plausible falsehoods, and offers no sources — so users can't tell right from wrong. An answer engine fixes this by grounding every answer in retrieved, citable sources.
Three fatal gaps: the model's knowledge is frozen at its training cutoff (no recent info), it hallucinates plausible falsehoods with total confidence, and it provides no sources to check. For a product whose job is factual answers, that's untrustworthy. You have to ground the answer in real, current, citable evidence — which the model doesn't have on its own.
Why this piece earns its place
The instinct on meeting this failure is to fix it with words — add only answer if you are certain to the system prompt and ship. It does almost nothing, and why it does nothing is the reason everything else on this board exists. The model carries no internal signal separating what it recalled from what it invented, so the instruction is addressed to a faculty nobody implemented. What it can measure is how probable its own wording was, and that is a statement about text rather than about the world: a well-worn falsehood scores higher than an awkwardly phrased truth, which is the opposite of the ordering you wanted. The repair has to be structural, not verbal. Two consequences worth carrying forward. Parametric memory does not switch off once retrieval is added — it stays underneath as a prior, and the answers hardest to catch downstream are the blended ones: three sentences from the passages and one from the weights, sitting under the same citation as though it had come from the same place. And the cutoff is not a clean edge. Training data thins toward it rather than stopping, so recent topics come back half-right, which reads far more credible than plainly wrong and is correspondingly harder to flag.
Step 2 · Retrieve, then generate
RAG over the live web
If the model can't be trusted to know the answer, where should the facts come from, and in what order do you do things?
Invert it: retrieve first, generate second. Take the question, find relevant, current sources (the web, or your document set), and feed those passages to the LLM as context, instructing it to answer only from them and cite each claim. The LLM becomes a reasoner/synthesizer over supplied evidence, not an oracle recalling from memory. This is retrieval-augmented generation applied to the open web — the core shape of an answer engine.
Why this piece earns its place
Inverting the order buys correctness, and it also changes which part of the system you are married to. While knowledge lives in weights, changing models means re-establishing everything the product knows; once it lives in retrieval, the model becomes the cheap component — swap it, re-run the evals, keep the corpus. That flips what you shop for in a model, too. Breadth of world knowledge is what a model launch advertises, and it is the one thing this design is deliberately suppressing; what you actually need is a model that stays inside the context it was handed and is willing to produce nothing. A smaller, more obedient model is often the better pick here, which is a budget decision you can act on the same afternoon. What the inversion does not fix is the reading. Retrieval moved the facts out of the weights; inference over those facts is still generation. So the sentences most likely to be wrong in a grounded answer are the ones that join two passages — a comparison, a total, a conclusion neither source states — because no single citation covers a join, and nothing later in this pipeline is looking at one.
What the new pieces do
- Answer Servicegateway
- Coordinates the pipeline: understand the query, retrieve sources, generate a grounded answer, verify, and stream it back.
Step 3 · The orchestrator
An answer service that streams
This is now a multi-step pipeline (understand → search → fetch → rank → generate → verify), and users won't wait many seconds staring at a blank screen. What runs the pipeline and keeps it responsive?
An answer service orchestrates the steps and streams the experience: show the interpreted query and the sources it's reading while retrieval runs, then stream the answer token-by-token with citations as it's generated. The multi-step latency (search + fetch + LLM) is masked by progressive UI — the user sees progress the whole time. The service also handles follow-up questions with conversation context (an answer engine is often conversational).
Why this piece earns its place
Streaming is sold as a perceived-speed trick, and it is one, but it also takes away your ability to change your mind. Once the first source card or the first token has left the server you have committed: no silent retry against a different model, no quiet re-run when a fetch comes back empty, no withdrawing a source the verification pass later distrusts. Every fallback you want has to be expressible as something appended to a stream already in flight — a correction, a struck citation, a visible note — rather than a do-over the user never sees. So decide early which parts of the stream are promises and which are revocable, because the UI encodes that choice permanently. The orchestrator also has to own the deadline. Every stage wants more time than the request has, and somebody must decide mid-flight to fetch fewer pages, or to ship while verification is still running. No individual stage can make that call, because none of them can see what the others have already spent.
- orchestratemulti-step pipeline
- streamsources then answer
- follow-upsconversational context
Step 4 · Ask the right thing
Query understanding & decomposition
The user's raw question is often a poor search query — vague, conversational, or actually several questions ("compare X and Y on price and safety"). Feeding it verbatim to a search engine retrieves weak sources. How do you get good evidence?
The raw question makes a poor search query (vague or multi-part). What do you do first?
A query planner (often an LLM) turns the messy question into one or more effective search queries — rewriting for clarity, expanding terms, and splitting a compound question into sub-questions it retrieves for separately, then combines. Better queries retrieve better sources, which is most of answer quality.
Verbatim conversational questions often retrieve poorly (they're not how documents are phrased, and compound questions need multiple searches). Rewriting and decomposition materially improve the evidence you get.
That's the bare-LLM failure again. The point is to retrieve good sources, which starts with turning the question into good queries.
Add a query planner (usually an LLM step) that rewrites the question into effective search queries — clarifying, expanding, adding context — and decomposes multi-part questions into sub-queries it retrieves for separately, then synthesizes together (multi-hop). Good retrieval starts with good queries, and this step is where a lot of answer quality is won or lost, before generation even begins.
Why this piece earns its place
The planner is the one stage whose mistakes are invisible to everything downstream, because when it goes wrong, nothing fails. A rewrite that drops a negation, resolves the wrong entity in a follow-up, or quietly narrows a comparison to one side produces healthy retrieval, confident generation, and citations that genuinely support what was asked — just not what the user asked. So log the planner output verbatim beside the answer. Without it a bad answer is undebuggable, because you cannot tell weak sources from a mangled question, and those have opposite fixes. It is also, quietly, the budget-setter. Decomposition turns one search into several, and each sub-query fans out into its own fetches, reranks and context, so a planner with a taste for splitting things multiplies the cost of an answer by however many sub-queries it feels like emitting, without anyone deciding to spend it. That makes the number of sub-queries a request-level knob, bounded by the orchestrator and informed by the question, rather than a constant in a config file tuned on whatever examples happened to be around that week.
- rewritequestion → good queries
- decomposesub-queries
- multi-hopretrieve then combine
What the new pieces do
- Query Plannerservice
- Turns a messy question into good search queries, decomposing multi-part questions into sub-queries (multi-hop).
Step 5 · Get the evidence
Search, fetch & rerank
With good queries, you retrieve candidate documents — but a search API returns links, pages are long and noisy, and the top search result isn't always the most relevant passage. How do you turn queries into a tight set of high-quality passages to ground on?
Run the queries against a search API (or your own index), then fetch the top pages, extract the main content (strip nav/ads), and chunk it. Then rerank the chunks by relevance to the question (a cross-encoder/reranker) — and weight freshness for time-sensitive questions — keeping only the best few passages that fit the LLM's context. You've gone from a vague question to a small, high-signal evidence set. Caching popular queries/pages cuts latency and cost.
Why this piece earns its place
Search and generation call vendors you hold a contract with. Fetch calls whoever the ranking happened to return — arbitrary third parties with no agreement, no capacity promise and no obligation to serve you at all — which makes it the one stage where partial failure is the normal case rather than the exception. Pages time out, sit behind paywalls, return a JavaScript shell with no text in it, or refuse your crawler outright. You cannot wait for all of them, so you need an explicit rule for what enough evidence means — a deadline, a minimum count, or both — and the moment that rule exists, the answer quietly depends on which pages happened to be fast. That correlation is not random: fast pages are CDN-hosted, lightly built and tuned for search, which is not the same population as the best sources. Nothing errors, no dashboard moves, and the engine develops a permanent taste for a certain kind of page. One more thing to carry: fetching at this volume makes you a crawler with a reputation, so identify yourself, honour rate limits, and cache hard — otherwise the sites you most want to ground on are the first to shut you out.
- searchwide recall
- fetch + extractclean passages
- rerankbest few + freshness
What the new pieces do
- Searchservice
- Runs the queries against a web search API or an internal index, returning candidate documents to ground on.
- Fetch + Rerankservice
- Fetches pages, extracts and chunks the text, and reranks passages by relevance (and freshness) to keep the best few.
- Sourcesdata
- The live corpus — the web (or your document set) — that provides current, attributable evidence for answers.
Step 6 · Answer from the sources
Grounded generation with citations
Now the LLM writes the answer. The whole value proposition is that it answers from the retrieved sources and attributes each claim — not that it free-associates. How do you make generation grounded and cited?
How do you make the LLM answer from the sources and attribute claims, not free-associate?
Merely providing sources isn't enough — the model may still lean on parametric memory or state unsupported claims. You must instruct it to answer only from the provided passages and to cite each claim to a specific source, and then verify.
Post-hoc citation (answer first, find sources later) invites citations that don't actually support the claim. Grounding must drive generation: answer FROM the sources, citing as you go.
Give the model the reranked passages with ids and instruct it to answer using only them, citing the source for every claim inline. Constraining generation to the evidence (and refusing when the sources don't cover it) is what produces grounded, attributable answers — then a verification step checks the citations.
Feed the LLM the reranked passages (with source ids) and prompt it to answer only from them, attaching an inline citation to every claim (and to say "the sources don't cover this" rather than guess). Grounding must drive generation — answer from the evidence, citing as you go — not be bolted on afterward. This is what turns an LLM into a trustworthy answer engine: every sentence traceable to a source.
Why this piece earns its place
Treat a citation as a data structure, not a marker in the text. A claim points at a passage id, and the passage points at a URL and the span inside it. Mint that id at extraction and carry it through rerank into the prompt, because once the answer exists you cannot recover from prose which passage the model meant, and a sentence naming a plausible-sounding source is a hallucination wearing a citation. Spans buy a second thing URLs cannot: they let you see when three citations are one story syndicated three times. Apparent corroboration is the most flattering way for a cited answer to be wrong, and at the URL level three domains look like three confirmations. Provenance has to be threaded through from extraction onward, because retrofitting it means refetching a page that may no longer say the same thing — which is the other half of this. The web moves under you. Between your fetch and the user clicking through, a page can be edited, paywalled or deleted, and a citation nobody can check is worth roughly what no citation is worth. So snapshot the extracted passage you actually used and keep it beside the live link. It also gives you something honest to say when the two disagree — that the page changed after the answer was written — instead of silently looking wrong.
- only from sourcesno free-association
- cite each claiminline attribution
- refusewhen unsupported
What the new pieces do
- Grounded LLMservice
- Synthesizes an answer using ONLY the retrieved passages, attaching an inline citation to each claim.
Step 7 · Trust, but verify
Citation verification & follow-ups
Even instructed to cite, an LLM can attach a citation that doesn't actually support the sentence, or subtly misstate a source. "Has citations" isn't "is grounded." How do you close that gap?
Add a verification step: check that each cited source actually supports its claim (an entailment/groundedness check, often another model pass), flagging or removing unsupported statements before the answer is finalized. Stream the verified answer with clickable citations, and support follow-up questions that reuse context and retrieve again. Optionally show a confidence signal and let users inspect the exact source passage. Verification is what upgrades citations from decoration to guarantee.
Why this piece earns its place
Three things you learn by running one. First, verification is a classifier with two error directions and different costs: a false flag strips a true sentence and makes the product look evasive, a miss ships an unsupported claim wearing a citation, the exact failure it existed to prevent. Which way you tune it is a product decision, not an engineering default. Second, do not let the model that wrote the answer check it; asked whether its own sentence follows from a passage it just used, it agrees, and you have bought a pass rate rather than a check. A separate, smaller entailment model is cheaper and less agreeable, and cheaper matters here for a reason: unit cost tracks answer length rather than question difficulty, so a verbose model raises the bill on every request, not just the hard ones. Third, it throws off the pipeline's most useful by-product: a log of claim, passage and verdict — labelled data nobody had to annotate. Flag rate sliced by domain beats any retrieval dashboard as an extraction alarm: when one site's claims start failing entailment and nothing else moves, the extractor broke there. And it only ever checks a claim against the passage that claim cites, so it is blind to a source that is confidently wrong, and blind to omission — an answer true sentence by sentence and misleading overall because the contradicting page never made the shortlist.
- verifysource supports claim?
- flagunsupported statements
- follow-upsre-retrieve with context
What the new pieces do
- Verifyservice
- Checks that each citation actually supports its claim (groundedness), catching unsupported or misattributed statements.
Step 8 · The sharp edges
Bad sources, injection & cost
The open web is hostile: sources contradict or lie, pages contain prompt injection aimed at your model, freshness varies, and every answer costs multiple model calls plus network fetches.
Judge source quality/trust (rank reputable domains higher, cross-check claims across sources, surface disagreement rather than picking one). Defend against prompt injection from fetched pages — treat retrieved web content as untrusted data, never instructions, and sandbox it so a page can't hijack the model. Handle freshness with recency-aware ranking and re-fetching. Control cost/latency (planner + rerank + generate + verify = several model calls plus fetches) with caching, parallel fetches, right-sized models per step, and skipping steps for simple queries. And be honest when the web simply doesn't have a good answer.
Why this piece earns its place
The injection defence people describe in interviews is usually a sentence in a prompt, and a sentence in a prompt is not a boundary. What actually holds is limiting what the model is able to do after it has read a stranger's page. Do not comfort yourself that the blast radius here is small. This product's whole job is to hand that page's words to a user as a cited claim, carrying the engine's credibility and a clickable link, so a page that wins the shortlist can choose where your reader goes next and which sources get cited beside it. And the moment you add a tool, a second fetch the model chooses, a memory write, a purchase, that same page inherits the capability. So fix the privilege of a request before retrieval runs, and never let anything read afterwards raise it. That one rule is what keeps this design safe to extend. The other trap is turning source trust into a domain allowlist because an allowlist is easy to implement. It is an editorial position shipped as a config file: it decides whose account of a contested question your users see, it rots without anyone noticing, and it is the artefact that gets screenshotted. A ranking weight you can defend and adjust is a very different thing to own than a gate.
You did it
You just designed an AI answer engine.
Everything you assembled, in order
- A bare LLM is stale, hallucinates, and cites nothing — untrustworthy for factual answers.
- Invert to retrieve-then-generate (RAG over the live web): facts from retrieval, reasoning from the LLM.
- An answer service orchestrates the pipeline and streams sources-then-answer to mask latency.
- A query planner rewrites and decomposes the question into good (multi-hop) search queries.
- Search wide, then fetch, extract, chunk and rerank (with freshness) into a tight evidence set.
- Generate grounded: answer only from the passages, citing each claim; refuse when unsupported.
- Verify that citations actually support their claims; judge source trust and defend against injection.