The big idea
What problem does GraphRAG solve?
Vanilla RAG retrieves the top-k most similar chunks and hands them to the LLM. That's great for a local lookup ("what's the refund policy?") — but ask something that needs connecting facts across many documents ("how are these three incidents related?") or a global question ("what are the main themes across this whole corpus?") and it fails: the answer isn't in any single chunk, it's in the relationships between them.
GraphRAG builds a knowledge graph (entities + relationships) from the corpus so retrieval can traverse connections, and clusters the graph into communities with pre-written summaries so global questions become a map-reduce over a few summaries. It adds structure and connectivity on top of RAG — answering the multi-hop and holistic questions similarity search can't.
How to read this: Each step opens with a real design decision — make the call before I show you what ships. Watch the pipeline grow, and at the end drop the graph and disable community summaries to see what breaks for connected and global questions. Hit Begin.
Step 1 · Where vector RAG breaks
Isolated chunks, no connections
Vector RAG embeds the query, finds the nearest chunks, and stuffs them into the prompt. Name the class of questions it fundamentally can't answer well, and why.
Design decision: Vector RAG retrieves the top-k most similar chunks. What can't it answer well?
The call: Simple factual lookups. — Those are exactly what vector RAG does well — a local fact usually lives in one similar chunk. Its weakness is connected/global questions, not simple lookups.
Vector RAG returns chunks that individually resemble the query — perfect for a local fact living in one chunk. It fails when the answer requires connecting information across many chunks: multi-hop reasoning ("A relates to C through B", where A, B, C are in different documents) or global synthesis ("the main themes across everything"). Those answers live in the relationships and structure of the corpus, which isolated similarity search never captures.
Similarity ≠ connection: Retrieving the k nearest chunks optimizes for local resemblance, not for the web of relationships between facts. Multi-hop and holistic questions are about structure — how things connect — so you need a representation of the corpus that encodes connections, not just points in embedding space.
Step 2 · Add structure
Represent the corpus as a graph
If the missing thing is connections, what representation captures them, and how does that change what retrieval can do?
Build a knowledge graph over the corpus: entities (people, orgs, concepts, events) as nodes and their relationships as edges. Now retrieval can traverse — start at the entities in the question and follow edges to connected facts, gathering a connected context instead of a bag of similar chunks. The graph turns "find similar text" into "follow the relationships", which is exactly what multi-hop questions need.
Nodes + edges = connections: A graph makes relationships first-class. Traversal reaches facts that aren't textually similar to the query but are connected to it (two hops away), which similarity search would never surface. This connectivity is the whole point — and it also enables the global summarization we'll add next.
Step 3 · Build the graph
Entity & relationship extraction
Where does the graph come from? Your corpus is unstructured text — you have to derive entities and relationships from it. How, and at what cost?
Design decision: Your corpus is plain text. How do you build a knowledge graph from it?
The call: Run an LLM (or NER model) over each chunk to extract entities and the relationships between them, merging into a graph. — At indexing time, an LLM reads each chunk and extracts entities and their relationships (subject–relation–object), which are merged (deduping/aliasing entities) into a single knowledge graph. It's an LLM pass per chunk — expensive up front — but it's what turns unstructured text into a traversable graph.
At indexing time, run an LLM (or NER/relation-extraction models) over each chunk to extract entities and the relationships between them (subject–relation–object), then merge them — deduping and aliasing the same entity across chunks — into one knowledge graph. This is the expensive part: an LLM pass per chunk. But it's what converts unstructured text into a connected, traversable structure that the query side can exploit.
Extraction is the indexing cost: GraphRAG front-loads work: you pay LLM extraction over the whole corpus at index time to build the graph. Entity resolution (merging "IBM" and "International Business Machines") matters — a messy graph traverses badly. The payoff is cheaper, richer answers to connected questions at query time.
Step 4 · Answer the whole corpus
Communities & hierarchical summaries
Traversal handles multi-hop, but a truly global question ("what are the major themes across all 10,000 documents?") can't be answered by traversing from a few entities — it's about the whole corpus. Reading everything per query is impossible. How do you enable global questions?
Detect communities in the graph (clusters of densely-connected entities, via graph clustering like Leiden), then summarize each community with an LLM — and do it hierarchically (summaries of summaries) up to the whole corpus. Now a global question is answered by map-reduce over community summaries: answer it against each relevant community summary (map), then combine those partial answers (reduce). You get a holistic answer without reading every document at query time.
Communities = pre-computed global structure: Community detection + summarization pre-computes the corpus's thematic structure at index time. A global query then reduces over a handful of summaries instead of the full corpus — the map-reduce pattern that makes "summarize everything" tractable. This is GraphRAG's signature capability over vanilla RAG.
Step 5 · Two ways to ask
Local vs global query modes
Different questions need different retrieval. How does the system decide how to gather context for a given query?
Design decision: How should retrieval differ for "what did entity X do?" vs "what are the overall themes?"
The call: Always traverse the entire graph. — Traversing everything is expensive and unnecessary for a local question, and doesn't give the pre-summarized holistic view a global question needs. Match the mode to the question.
Two modes. Local (entity-centric): find the entities in the query, traverse the graph around them to gather connected facts (and the chunks backing them) — for specific, multi-hop questions. Global (thematic): map-reduce over community summaries — for holistic, "across everything" questions. The system routes each query to the right mode (or blends), assembling a connected context the LLM then synthesizes into a cited answer.
Route to the mode: Local = traverse from entities (precision, multi-hop). Global = reduce over community summaries (breadth, themes). Picking the mode (often via a lightweight classifier or the agent deciding) is how GraphRAG serves both narrow and broad questions from the same index.
Step 6 · Best of both
Hybrid graph + vector retrieval
The graph is powerful for connections, but plain vector search is still great for finding the specific passage that states a fact. Should you throw away vector RAG? No. How do they combine?
Run hybrid retrieval: use the graph for structure and connections (traversal, communities) and keep vector search over chunks for pinpoint semantic matches, then fuse them into one context — graph-derived connected facts plus the exact supporting chunks (which also give you citations). The LLM gets both the "how things connect" from the graph and the "here's the passage that proves it" from vectors, producing answers that are connected and grounded.
Graph + vectors, not either/or: GraphRAG augments rather than replaces vector RAG. Vectors excel at local semantic retrieval and provide citable source chunks; the graph adds connectivity and global structure. Combining them gives multi-hop/global reasoning with precise, attributable evidence — the strongest form.
Step 7 · Keeping it current
Indexing cost & incremental updates
Building the graph (LLM extraction per chunk + community summarization) is expensive and slow, and corpora change — new documents arrive constantly. Re-building the whole graph each time is untenable. How do you keep it fresh affordably?
Treat graph construction as an expensive index build you amortize, and support incremental updates: when new documents arrive, extract their entities/relationships and merge into the existing graph (re-resolving entities), and re-summarize only the affected communities rather than the whole corpus. Cache and version the graph and summaries. Accept that indexing is the cost center of GraphRAG — you pay more at index time to get better answers to hard questions at query time.
Amortize + update incrementally: The graph is a heavy, precomputed asset. Incremental extraction + localized re-summarization keep it current without full rebuilds. The economic model is the inverse of vanilla RAG: higher indexing cost, unlocking answers to connected/global questions that were previously impossible — worth it only when those questions matter.
Step 8 · The sharp edges
Extraction errors, cost & when to use it
GraphRAG has real downsides: LLM extraction makes mistakes (wrong/missing entities and relations), indexing is costly, and it's overkill for many workloads.
Manage extraction quality — errors propagate into a wrong graph, so use good prompts/models, entity resolution, and validation, and remember the graph is an approximation. Control cost (LLM per chunk + summarization) by scoping GraphRAG to corpora and question types that need it. Crucially, know when to use it: for mostly-local factual Q&A, vanilla vector RAG is cheaper and sufficient — GraphRAG earns its cost only when multi-hop or global/thematic questions matter (research synthesis, investigations, large connected corpora). Default to hybrid, and reach for the full graph when the questions demand connectivity.
Design for the unhappy path: Extraction errors → validate, resolve entities, expect approximation. Cost → scope it to where it pays. Overkill → use vanilla RAG for local Q&A; add the graph for connected/global questions. GraphRAG is a powerful, expensive upgrade — apply it deliberately, not by default.
You did it
You just designed a GraphRAG system.
- Vanilla vector RAG retrieves isolated similar chunks — it fails multi-hop and global questions.
- Build a knowledge graph (entities + relationships) so retrieval can traverse connections.
- Construct the graph by LLM-extracting entities/relations per chunk and merging (entity resolution).
- Detect communities and write hierarchical summaries — this enables global/thematic questions.
- Route queries: local (traverse from entities) vs global (map-reduce over community summaries).
- Use hybrid graph + vector retrieval for connected reasoning with precise, citable evidence.
- Indexing is the cost center — amortize it, update incrementally, and use GraphRAG only where it pays.