The whole design, in writing
Learn AI system design by building a GraphRAG system — knowledge-graph-augmented retrieval — step by step. An interactive guide covering why vanilla vector RAG fails on global and multi-hop questions, building a knowledge graph by extracting entities and relationships, community detection and hierarchical summaries, local vs global query modes, hybrid graph+vector retrieval, indexing cost, and the unhappy paths (extraction errors, cost, when a graph helps).
Every step of the build above, written out: the problem each piece solves, the option that was taken and the ones that were not, the numbers, and how it fails in production.
The big idea
What problem does GraphRAG solve?
Vanilla RAG retrieves the top-k most similar chunks and hands them to the LLM. That's great for a local lookup ("what's the refund policy?") — but ask something that needs connecting facts across many documents ("how are these three incidents related?") or a global question ("what are the main themes across this whole corpus?") and it fails: the answer isn't in any single chunk, it's in the relationships between them.
GraphRAG builds a knowledge graph (entities + relationships) from the corpus so retrieval can traverse connections, and clusters the graph into communities with pre-written summaries so global questions become a map-reduce over a few summaries. It adds structure and connectivity on top of RAG — answering the multi-hop and holistic questions similarity search can't.
What the new pieces do
- Clientclient
- Asks anything from a narrow fact ("what did X say?") to a global one ("what are the main themes across all these docs?").
Step 1 · Where vector RAG breaks
Isolated chunks, no connections
Vector RAG embeds the query, finds the nearest chunks, and stuffs them into the prompt. Name the class of questions it fundamentally can't answer well, and why.
Vector RAG retrieves the top-k most similar chunks. What can't it answer well?
Similarity search returns chunks that individually resemble the query, but questions like "how does A connect to C via B?" (multi-hop) or "what are the overall themes?" (global) require reasoning over connections spanning many documents. No single chunk contains that answer, and retrieving a handful of similar chunks misses the structure — you need the relationships, which vanilla RAG doesn't model.
Those are exactly what vector RAG does well — a local fact usually lives in one similar chunk. Its weakness is connected/global questions, not simple lookups.
Language is a separate concern (handled by multilingual embeddings). The structural gap is connected/multi-hop and global reasoning across documents.
Vector RAG returns chunks that individually resemble the query — perfect for a local fact living in one chunk. It fails when the answer requires connecting information across many chunks: multi-hop reasoning ("A relates to C through B", where A, B, C are in different documents) or global synthesis ("the main themes across everything"). Those answers live in the relationships and structure of the corpus, which isolated similarity search never captures.
Why this piece earns its place
The reason this deserves its own step is that the failure never reaches your telemetry. A multi-hop question does not error and does not time out — the retriever returns exactly k chunks with ordinary-looking scores, the model writes a fluent paragraph from them, and the only thing wrong is that the paragraph is thin. The eval harness is blind to it too: retrieval recall is scored against a known-correct chunk, and a question whose answer spans three documents has no single chunk to label, so the metric stays green while the answers get worse. The demand signal is suppressed as well. People learn quickly what a tool is bad at, so after a few vague replies they stop asking connected questions altogether, and the query log then under-reports exactly the traffic that would justify building a graph. So sample real questions by hand — from support threads, from the people who gave up, not from the log — and sort them into local, multi-hop and global. Keep that sample afterwards, because it is also the only before measurement you will ever get. Once the graph ships, the traffic changes shape: people discover the system can follow a chain and start asking things they never bothered to type, and a query log written by users who had already learned what not to ask cannot be the control group.
Step 2 · Add structure
Represent the corpus as a graph
If the missing thing is connections, what representation captures them, and how does that change what retrieval can do?
Build a knowledge graph over the corpus: entities (people, orgs, concepts, events) as nodes and their relationships as edges. Now retrieval can traverse — start at the entities in the question and follow edges to connected facts, gathering a connected context instead of a bag of similar chunks. The graph turns "find similar text" into "follow the relationships", which is exactly what multi-hop questions need.
Why this piece earns its place
The decision hiding inside this step is the schema: what counts as an entity, and which relationship labels are allowed. Fully open extraction — keep whatever the model finds — produces a graph whose node types are as varied as the prose that made them and whose edges are phrased in free text, where works_for, employed by and is staff at are three unrelated edges and no traversal can ask for all three. A closed schema traverses cleanly and silently discards anything it did not anticipate. Most real systems settle on a small fixed set of relationship types with the original wording carried alongside for display. Choose deliberately, because the schema is baked into every extraction you have already paid for: widening it later means re-reading the corpus, while adding another node or edge under the existing schema is just a write. The second thing the graph buys you is a doorway for data that was never prose. An org chart, a ticket system’s foreign keys, a product catalogue — those are already entities and relationships, so they merge into the same graph with no model in the loop and no extraction error to inherit.
What the new pieces do
- GraphRAG Svcgateway
- At index time builds the graph; at query time picks local vs global mode and assembles context for the LLM.
Step 3 · Build the graph
Entity & relationship extraction
Where does the graph come from? Your corpus is unstructured text — you have to derive entities and relationships from it. How, and at what cost?
Your corpus is plain text. How do you build a knowledge graph from it?
Manual graph construction doesn't scale to a real corpus. You extract the graph automatically from the text.
Titles are far too coarse — the entities and relationships live inside the text. You need to extract them from the chunk content.
At indexing time, an LLM reads each chunk and extracts entities and their relationships (subject–relation–object), which are merged (deduping/aliasing entities) into a single knowledge graph. It's an LLM pass per chunk — expensive up front — but it's what turns unstructured text into a traversable graph.
At indexing time, run an LLM (or NER/relation-extraction models) over each chunk to extract entities and the relationships between them (subject–relation–object), then merge them — deduping and aliasing the same entity across chunks — into one knowledge graph. This is the expensive part: an LLM pass per chunk. But it's what converts unstructured text into a connected, traversable structure that the query side can exploit.
Why this piece earns its place
Two things about this pass only show up once you have run it. First, the blast radius of a mistake here is nothing like a mistake at query time. A bad retrieval spoils one answer; a bad extraction is written into the graph and quietly shapes every answer that ever traverses that edge, until somebody re-extracts. That is the argument against economising on the model here, which is otherwise the most tempting place to do it: a weaker extractor does not buy you a slightly worse answer, it buys you a permanently worse asset, and the saving is banked once while the damage is read back forever. Spend here and economise somewhere you can change your mind later. Second, make the pass resumable and idempotent before you start it rather than after. A pass over a whole corpus runs long enough to be interrupted — a rate limit, a deploy, one chunk that crashes the worker — and if the rerun does not skip chunks already done you pay the entire bill twice. Key the output by chunk id so a rerun is a no-op over finished work, and keep the raw triples as their own durable artifact, separate from the merged graph, so you can re-run the merge under better resolution rules without re-reading a single document.
- LLM per chunkextract entities + rels
- mergeentity resolution
- index-time costbuild the graph once
What the new pieces do
- Extractorservice
- Indexing step: an LLM reads each chunk and extracts entities and the relationships between them.
- Knowledge Graphdata
- The extracted graph: entities as nodes, relationships as edges — a connected map of the corpus, not isolated chunks.
Step 4 · Answer the whole corpus
Communities & hierarchical summaries
Traversal handles multi-hop, but a truly global question ("what are the major themes across all 10,000 documents?") can't be answered by traversing from a few entities — it's about the whole corpus. Reading everything per query is impossible. How do you enable global questions?
Detect communities in the graph (clusters of densely-connected entities, via graph clustering like Leiden), then summarize each community with an LLM — and do it hierarchically (summaries of summaries) up to the whole corpus. Now a global question is answered by map-reduce over community summaries: answer it against each relevant community summary (map), then combine those partial answers (reduce). You get a holistic answer without reading every document at query time.
Why this piece earns its place
The thing to know about community detection is that the partition is not stable. Add documents, re-cluster, and the algorithm can redraw boundaries across the whole graph — one community splits, two merge, the numbering shifts. Community ids are therefore not durable identifiers, and anything treating them as such breaks quietly: a cached global answer, a saved link into a summary, a dashboard counting summaries per theme. Give each summary an identity derived from its members, or accept that everything downstream is re-derived on every re-clustering. The second point is that a summary is derived state three removes from the source — text, then graph, then clusters, then prose — so each of those can go stale on its own, and the errors compound without ever contradicting each other. A summary written from last month’s partition reads perfectly well and describes a grouping that no longer exists, which is the hardest kind of wrong to notice: fluency is not evidence of freshness, and nothing in the prose itself gives away its age. And treat the hierarchy level as a per-request choice: reducing over the top level is cheap and vague, reducing over the leaves is slow and specific, and that knob belongs in the query, not the config.
- communitiesgraph clusters
- summariesper community, hierarchical
- global = map-reduceover summaries
What the new pieces do
- Communitiesservice
- Detects clusters (communities) in the graph and writes a summary of each, hierarchically — the key to global questions.
Step 5 · Two ways to ask
Local vs global query modes
Different questions need different retrieval. How does the system decide how to gather context for a given query?
How should retrieval differ for "what did entity X do?" vs "what are the overall themes?"
Entity-centric (local) questions start at the mentioned entities and traverse their neighborhood in the graph to gather connected facts. Global/thematic questions use the community summaries via map-reduce. The system routes to the mode that fits the question (and can blend them).
Traversing everything is expensive and unnecessary for a local question, and doesn't give the pre-summarized holistic view a global question needs. Match the mode to the question.
Community summaries are for global questions; a precise local fact is better answered by traversing from the specific entity. You need both modes.
Two modes. Local (entity-centric): find the entities in the query, traverse the graph around them to gather connected facts (and the chunks backing them) — for specific, multi-hop questions. Global (thematic): map-reduce over community summaries — for holistic, "across everything" questions. The system routes each query to the right mode (or blends), assembling a connected context the LLM then synthesizes into a cited answer.
Why this piece earns its place
Routing looks like a classification problem and is mostly a budget problem. The two modes do not cost remotely the same: a local question is served out of one neighbourhood, while a global one spends a model call on every community summary it maps over before it can reduce them. So a misroute to global is not a slightly worse answer, it is a request whose price is set by how many communities the graph happens to contain — a number the person asking neither sees nor controls, and one that grows with the corpus. Any user can trigger it over and over just by phrasing things broadly. Global mode needs its own concurrency cap and its own spend limit, separate from the service’s overall rate limit. The quieter failure lives inside local mode. Before it can traverse, it has to match the names in the question to nodes in the graph — the same resolution problem as indexing, now on the critical path with no time to deliberate. If the question names something the graph canonicalised differently, traversal starts from nowhere and returns an empty neighbourhood, and the answer reports that there is no information rather than admitting it could not find the entity. Log the linked entities on every query; an empty link set is the signal, and nothing else exposes it.
- localtraverse from entities
- globalmap-reduce summaries
- routemode per question
What the new pieces do
- Retrieverservice
- Query time: traverses the graph (local) or map-reduces community summaries (global), fused with vector chunks.
- LLMservice
- Reads the assembled graph + chunk context and writes the grounded, cited answer.
Step 6 · Best of both
Hybrid graph + vector retrieval
The graph is powerful for connections, but plain vector search is still great for finding the specific passage that states a fact. Should you throw away vector RAG? No. How do they combine?
Run hybrid retrieval: use the graph for structure and connections (traversal, communities) and keep vector search over chunks for pinpoint semantic matches, then fuse them into one context — graph-derived connected facts plus the exact supporting chunks (which also give you citations). The LLM gets both the "how things connect" from the graph and the "here's the passage that proves it" from vectors, producing answers that are connected and grounded.
Why this piece earns its place
Fusing two retrievers sounds like concatenation and is really a ranking problem, because the two scores are not comparable. Traversal gives you hop distance and edge weights; vector search gives you a similarity. No threshold means the same thing on both, so a merged list cannot simply be sorted by score. Either fuse at the level of rank rather than score, or put a single reranker over the union and let one model produce one ordering. The lazy version — take a fixed few from each side — is a budget split hardcoded in advance, generous to the graph on a simple lookup and stingy on a genuine multi-hop chain. The operational cost of keeping both indexes is that the corpus now has two write paths that can disagree. Delete a document, remove its chunks but not its edges, and you get an answer whose reasoning still holds and whose evidence has vanished — a claim with a dangling citation, which is worse than no citation because it looks checked. Decide once which store is authoritative for deletion, and reconcile the other toward it on a schedule.
- graphconnections + global
- vectorspinpoint + citations
- fuseconnected AND grounded
What the new pieces do
- Vector Chunksdata
- The classic chunk embeddings, kept alongside the graph so retrieval can combine graph structure with semantic similarity.
Step 7 · Keeping it current
Indexing cost & incremental updates
Building the graph (LLM extraction per chunk + community summarization) is expensive and slow, and corpora change — new documents arrive constantly. Re-building the whole graph each time is untenable. How do you keep it fresh affordably?
Treat graph construction as an expensive index build you amortize, and support incremental updates: when new documents arrive, extract their entities/relationships and merge into the existing graph (re-resolving entities), and re-summarize only the affected communities rather than the whole corpus. Cache and version the graph and summaries. Accept that indexing is the cost center of GraphRAG — you pay more at index time to get better answers to hard questions at query time.
Why this piece earns its place
What makes incremental updates genuinely hard is that a new document can change the meaning of old ones. A page establishing that two names refer to the same person retroactively merges nodes that were separate, rewiring edges written months earlier and moving entities between communities. The affected set is not the new document’s neighbourhood — it ripples, and the ripple has no clean bound. So the honest design runs both mechanisms: incremental merges for freshness, plus a scheduled full rebuild as the correctness backstop, with the rebuild producing a new graph version swapped in atomically rather than mutating the live one. Mutating in place means a query in flight traverses a half-updated graph and follows a hop that has disappeared by the time it fetches the supporting text. Pin the extractor model and prompt version to the graph version while you are there, or a rebuild meant only to add documents silently re-litigates every old edge under a different model and nobody can attribute the changed answers. That version is also the only unit a rollback can work in: you cannot un-merge a graph in place, so when a rebuild leaves answers worse than it found them, the recovery is to point traffic back at the previous version — which means keeping it, and its summaries, alive longer than it takes somebody to notice.
- index buildexpensive, amortized
- incrementalmerge new docs
- re-summarizeaffected communities only
Step 8 · The sharp edges
Extraction errors, cost & when to use it
GraphRAG has real downsides: LLM extraction makes mistakes (wrong/missing entities and relations), indexing is costly, and it's overkill for many workloads.
Manage extraction quality — errors propagate into a wrong graph, so use good prompts/models, entity resolution, and validation, and remember the graph is an approximation. Control cost (LLM per chunk + summarization) by scoping GraphRAG to corpora and question types that need it. Crucially, know when to use it: for mostly-local factual Q&A, vanilla vector RAG is cheaper and sufficient — GraphRAG earns its cost only when multi-hop or global/thematic questions matter (research synthesis, investigations, large connected corpora). Default to hybrid, and reach for the full graph when the questions demand connectivity.
Why this piece earns its place
The shape of the cost matters as much as its size. Vanilla RAG bills you for what people use; a graph bills you for what you own. Extraction runs over the corpus whether or not anybody ever asks a connected question, so an unused deployment costs about what a busy one does, and the money is committed before a single query has tested the premise. That should reorder the work: extract one representative slice of the corpus, see whether the answers off it are actually better, and only then hand over the archive to a pass you cannot un-spend. The second thing a single quality figure hides is that extraction is not uniformly good. Clean narrative prose hands a model the subject-relation-object shape it is looking for; transcripts, forms, tables and anything thick with pronouns hand it much less — and those are frequently the documents carrying the connections you built this for. Break the number out by source type, and expect a graph that is strongest where you needed it least. Then name an owner: retrieval treats the graph as an input, ingestion treats it as an output, and its quality belongs to neither of them until an answer embarrasses somebody.
You did it
You just designed a GraphRAG system.
Everything you assembled, in order
- Vanilla vector RAG retrieves isolated similar chunks — it fails multi-hop and global questions.
- Build a knowledge graph (entities + relationships) so retrieval can traverse connections.
- Construct the graph by LLM-extracting entities/relations per chunk and merging (entity resolution).
- Detect communities and write hierarchical summaries — this enables global/thematic questions.
- Route queries: local (traverse from entities) vs global (map-reduce over community summaries).
- Use hybrid graph + vector retrieval for connected reasoning with precise, citable evidence.
- Indexing is the cost center — amortize it, update incrementally, and use GraphRAG only where it pays.
