The big idea
What is an AI answer engine?
You ask a real question — "what changed in the latest React release?", "compare these two drugs' side effects". A plain chatbot will answer confidently from memory that's frozen at its training cutoff, sometimes right, sometimes hallucinated, and never with a source you can check. For questions where being current and correct matters, that's not good enough.
An AI answer engine (like Perplexity) answers by retrieving live sources first, then generating a grounded, cited answer from them — RAG over the open web. The pipeline: understand the question, search + fetch + rerank real sources, generate an answer using only those sources with inline citations, and verify the citations. Grounding and attribution are the whole point.
How to read this: Each step opens with a real design decision — make the call before I show you what ships. Watch the pipeline grow, and at the end cut off retrieval and disable citation checks to see why sources and verification are the product. Hit Begin.
Step 1 · Why not just ask the model?
The bare LLM fails
The simplest design: forward the question straight to an LLM and return its answer. Why is that inadequate for a question-answering product people should trust?
Design decision: Just send the question to an LLM and return its answer. What's wrong?
The call: Its knowledge is frozen at training time, it hallucinates confidently, and it cites nothing to verify. — A bare LLM can't know recent events, will sometimes fabricate plausible falsehoods, and offers no sources — so users can't tell right from wrong. An answer engine fixes this by grounding every answer in retrieved, citable sources.
Three fatal gaps: the model's knowledge is frozen at its training cutoff (no recent info), it hallucinates plausible falsehoods with total confidence, and it provides no sources to check. For a product whose job is factual answers, that's untrustworthy. You have to ground the answer in real, current, citable evidence — which the model doesn't have on its own.
Parametric memory isn't enough: An LLM's knowledge lives in its weights — parametric memory — which is stale, lossy, and unattributable. For current, verifiable answers you supply the facts at query time from an external, up-to-date source and make the model reason over those. That's retrieval-augmented generation.
Step 2 · Retrieve, then generate
RAG over the live web
If the model can't be trusted to know the answer, where should the facts come from, and in what order do you do things?
Invert it: retrieve first, generate second. Take the question, find relevant, current sources (the web, or your document set), and feed those passages to the LLM as context, instructing it to answer only from them and cite each claim. The LLM becomes a reasoner/synthesizer over supplied evidence, not an oracle recalling from memory. This is retrieval-augmented generation applied to the open web — the core shape of an answer engine.
RAG: facts in, reasoning out: Separate knowledge (retrieved, fresh, citable) from reasoning (the LLM). The model's job shrinks to "read these sources and answer the question, with citations" — which it's good at — while correctness and freshness come from retrieval, which you control.
Step 3 · The orchestrator
An answer service that streams
This is now a multi-step pipeline (understand → search → fetch → rank → generate → verify), and users won't wait many seconds staring at a blank screen. What runs the pipeline and keeps it responsive?
An answer service orchestrates the steps and streams the experience: show the interpreted query and the sources it's reading while retrieval runs, then stream the answer token-by-token with citations as it's generated. The multi-step latency (search + fetch + LLM) is masked by progressive UI — the user sees progress the whole time. The service also handles follow-up questions with conversation context (an answer engine is often conversational).
Stream the pipeline: A RAG-over-web answer takes real time (network round-trips to search and fetch pages, then generation). You hide that latency by streaming: surface sources first, then the grounded answer incrementally. Perceived speed comes from showing work, not from being instant.
Step 4 · Ask the right thing
Query understanding & decomposition
The user's raw question is often a poor search query — vague, conversational, or actually several questions ("compare X and Y on price and safety"). Feeding it verbatim to a search engine retrieves weak sources. How do you get good evidence?
Design decision: The raw question makes a poor search query (vague or multi-part). What do you do first?
The call: Send the exact question text to the search engine. — Verbatim conversational questions often retrieve poorly (they're not how documents are phrased, and compound questions need multiple searches). Rewriting and decomposition materially improve the evidence you get.
Add a query planner (usually an LLM step) that rewrites the question into effective search queries — clarifying, expanding, adding context — and decomposes multi-part questions into sub-queries it retrieves for separately, then synthesizes together (multi-hop). Good retrieval starts with good queries, and this step is where a lot of answer quality is won or lost, before generation even begins.
Query rewriting + multi-hop: The gap between how people ask and how documents are written is real; rewriting bridges it. Complex questions need several retrievals (decomposition / multi-hop) whose results are combined. Investing here beats trying to fix bad sources at generation time.
Step 5 · Get the evidence
Search, fetch & rerank
With good queries, you retrieve candidate documents — but a search API returns links, pages are long and noisy, and the top search result isn't always the most relevant passage. How do you turn queries into a tight set of high-quality passages to ground on?
Run the queries against a search API (or your own index), then fetch the top pages, extract the main content (strip nav/ads), and chunk it. Then rerank the chunks by relevance to the question (a cross-encoder/reranker) — and weight freshness for time-sensitive questions — keeping only the best few passages that fit the LLM's context. You've gone from a vague question to a small, high-signal evidence set. Caching popular queries/pages cuts latency and cost.
Retrieve wide, rerank narrow: Search casts a wide net (recall); reranking picks the few best passages (precision). Fetching and extracting clean text matters — feeding the model nav bars and ads wastes context and invites errors. Freshness ranking is what makes an answer engine current, not just relevant.
Step 6 · Answer from the sources
Grounded generation with citations
Now the LLM writes the answer. The whole value proposition is that it answers from the retrieved sources and attributes each claim — not that it free-associates. How do you make generation grounded and cited?
Design decision: How do you make the LLM answer from the sources and attribute claims, not free-associate?
The call: Prompt it to answer only from the provided passages and attach an inline citation to each claim, then verify. — Give the model the reranked passages with ids and instruct it to answer using only them, citing the source for every claim inline. Constraining generation to the evidence (and refusing when the sources don't cover it) is what produces grounded, attributable answers — then a verification step checks the citations.
Feed the LLM the reranked passages (with source ids) and prompt it to answer only from them, attaching an inline citation to every claim (and to say "the sources don't cover this" rather than guess). Grounding must drive generation — answer from the evidence, citing as you go — not be bolted on afterward. This is what turns an LLM into a trustworthy answer engine: every sentence traceable to a source.
Grounded, attributable generation: The prompt constrains the model to the supplied evidence and demands citations per claim. Refusing when evidence is missing is a feature, not a bug — it's the difference between "I don't have a source for that" and a confident hallucination. Citations make the answer checkable.
Step 7 · Trust, but verify
Citation verification & follow-ups
Even instructed to cite, an LLM can attach a citation that doesn't actually support the sentence, or subtly misstate a source. "Has citations" isn't "is grounded." How do you close that gap?
Add a verification step: check that each cited source actually supports its claim (an entailment/groundedness check, often another model pass), flagging or removing unsupported statements before the answer is finalized. Stream the verified answer with clickable citations, and support follow-up questions that reuse context and retrieve again. Optionally show a confidence signal and let users inspect the exact source passage. Verification is what upgrades citations from decoration to guarantee.
Groundedness check: Verification asks, per claim, "does the cited passage entail this?" — catching the model citing a real page that doesn't say what the sentence claims. It's the same idea as an LLM-as-judge groundedness eval, run inline. This is the difference between an answer that looks sourced and one that is.
Step 8 · The sharp edges
Bad sources, injection & cost
The open web is hostile: sources contradict or lie, pages contain prompt injection aimed at your model, freshness varies, and every answer costs multiple model calls plus network fetches.
Judge source quality/trust (rank reputable domains higher, cross-check claims across sources, surface disagreement rather than picking one). Defend against prompt injection from fetched pages — treat retrieved web content as untrusted data, never instructions, and sandbox it so a page can't hijack the model. Handle freshness with recency-aware ranking and re-fetching. Control cost/latency (planner + rerank + generate + verify = several model calls plus fetches) with caching, parallel fetches, right-sized models per step, and skipping steps for simple queries. And be honest when the web simply doesn't have a good answer.
Design for the unhappy path: Contradictory sources → cross-check + show disagreement. Malicious pages → retrieved content is data, not commands (injection defense). Stale info → recency ranking. Many model calls → cache, parallelize, right-size. The open web makes an answer engine powerful and adversarial — grounding must come with skepticism.
You did it
You just designed an AI answer engine.
- A bare LLM is stale, hallucinates, and cites nothing — untrustworthy for factual answers.
- Invert to retrieve-then-generate (RAG over the live web): facts from retrieval, reasoning from the LLM.
- An answer service orchestrates the pipeline and streams sources-then-answer to mask latency.
- A query planner rewrites and decomposes the question into good (multi-hop) search queries.
- Search wide, then fetch, extract, chunk and rerank (with freshness) into a tight evidence set.
- Generate grounded: answer only from the passages, citing each claim; refuse when unsupported.
- Verify that citations actually support their claims; judge source trust and defend against injection.