Graph RAG: Answering Questions Vector Search Structurally Can't

A procurement lead asks your internal assistant a reasonable question: “Which of our suppliers also supply Northwind?” Every supplier contract is indexed. So is the competitor research folder, which contains a memo naming Northwind’s component sources. Both halves of the answer are in the corpus. The assistant says it cannot find anything.

So you do the obvious things. Add a reranker. Raise top-k from five to fifty. Split the query into sub-questions. The answer still does not come out — or it comes out occasionally and you cannot tell why.

That is not a tuning problem. It is a shape problem.

Why can’t a better reranker fix this?

Similarity search ranks passages by how much they look like the question. That works beautifully when the answer is written down in words resembling the asking. It fails structurally in three situations.

Multi-hop questions. The supplier question needs two facts: we buy casings from Delta Polymers, and Delta Polymers supplies Northwind. The first passage resembles the question. The second does not — it never mentions us and may never use the word “supplier”. It resembles the intermediate answer, which you do not know yet at the moment you search. A reranker reorders the shortlist the retriever produced. It cannot promote a passage that was never on it.

Global or aggregate questions. “What are the recurring themes in last quarter’s support tickets?” has no correct answer in any chunk. It is a property of eleven thousand chunks taken together. Retrieve the top twenty and you have a sample, not a summary — and the model will write a confident thematic analysis of those twenty as though it had read all of them.

Relationship questions. “How does this component relate to that product line?” asks about an edge, not a document. The relationship may be stated nowhere and implied everywhere: the part appears in three bills of materials, two of which belong to that line. No passage says the thing you want to know.

Both halves use the same corpus. One side searches text; the other follows connections that were pulled out of that text ahead of time. If you have not read the overview of RAG and its variants, start there — Graph RAG makes much more sense as the answer to a specific failure than as an architecture in its own right.

What is a knowledge graph, in plain terms?

Strip away the ontology vocabulary and it is simple. Things become nodes: companies, products, SKUs, people, components, tickets. The facts connecting them become edges: supplies, is_a_part_of, replaced_by, applies_to. Each edge keeps a pointer back to the sentence it came from, which is what makes the whole thing citable.

The difference from the knowledge graphs of ten years ago is that nobody hand-builds this one. You do not hire an ontologist for six months; you run your documents through an LLM and ask it to pull out the entities and relations it sees, in a schema you specify. That is what made Graph RAG practical for ordinary companies rather than only Google-sized ones.

It also means the graph inherits every weakness of the extraction step. A model reading a messy contract will invent an edge occasionally, miss another, and name the same relationship three ways across three documents. The graph is not ground truth. It is a derived, noisy index — useful, but never a system of record.

How is the pipeline different from ordinary RAG?

Standard RAG has an offline half (chunk, embed, store) and an online half (embed the question, search, answer). Graph RAG keeps both and adds a much heavier offline stage in front, plus a different move at query time.

Extraction is the part people demo. Entity resolution is the part that decides whether the system works. If “Acme Ltd”, “Acme Limited” and “ACME (UK)” become three nodes, your two-hop path is broken and the query returns nothing — the same silent failure you were escaping, now with more infrastructure behind it. Doing it properly means normalising names, comparing aliases and embeddings, and using real identifiers wherever you have them: registration numbers, SKU codes and part numbers beat fuzzy string matching every time. Some merges will still need a human.

At query time the work changes character. Rather than only ranking passages, the system works out which entities the question is about, finds their nodes, and walks outward — one hop, two hops, along particular edge types — collecting the subgraph and the source passages hanging off it. That subgraph goes into the prompt. The model reasons over a small connected set of facts instead of a pile of independently plausible paragraphs.

How do you answer “what are the recurring themes?”

Traversal solves multi-hop. It does nothing for the global question, because there is no starting entity — the question is about everything.

The usual approach exploits structure the graph already has. Densely connected entities tend to be about the same thing: a knot of tickets, part numbers and error codes all revolving around one faulty component. Community detection algorithms find those clusters mechanically, without anyone declaring the topics in advance. Then, offline, you write a short summary of each cluster — and summaries of groups of clusters above that, giving a shallow hierarchy from “this specific failure mode” up to “returns and logistics”.

Now the corpus-wide question has something to retrieve. Instead of sampling twenty tickets, you pull the relevant cluster summaries, each already reflecting hundreds of documents. They are only as good as the extraction beneath them, but this turns an impossible question into an ordinary retrieval problem. All of it is model work done in advance across the whole corpus. Budget for it as a batch job, not a feature flag.

Should the graph replace your vector index?

No. Almost never. This is the part that gets lost in the enthusiasm.

A graph is excellent at structure and poor at nuance. It knows Delta Polymers supplies Northwind; it does not hold the three paragraphs of contractual wording on early termination that the user needs to read. The prose lives in the documents. So the architecture that ships keeps both: the vector index retrieves passages as it always did, the graph supplies connections and entity context, and the two result sets merge before the model sees anything.

Most teams end up routing by question type — a lookup goes to the vector index, a relational or thematic question triggers traversal, plenty of questions use both. That routing belongs in a modular RAG architecture, where retrieval strategies are swappable components rather than one hard-wired path.

What does it cost, and when should you not build one?

Be clear-eyed here. This is the most expensive type in the series, and the sales pitch rarely mentions the maintenance.

Extraction costs real money. Every chunk of every document goes through a model, more than once if you are also building community summaries. On a small policy corpus that is trivial. On a decade of contracts, tickets and product data it is a line item somebody has to approve — and you pay a fraction of it again each time you change the schema or improve the extraction prompt.

Entity resolution is genuinely hard. Not fiddly-hard. Hard as in an open industry problem, with real duplicates, real ambiguity (two unrelated Acmes), and a wrong merge doing more damage than no merge at all.

The graph has to stay in sync. A contract gets amended, a supplier is acquired, a part is superseded. Re-extracting everything nightly is wasteful; incremental updates mean working out which nodes and edges a changed document touched and retiring the ones it no longer supports. Stale edges are the nastiest failure of the lot, because they look authoritative — the system will confidently trace a path through a relationship that ended last year.

So here is the test we apply before recommending it. Write down the ten questions this system exists to answer — the ones with money or risk attached. Mark each: is it a lookup (the answer is written down in one place) or is it relational (the answer is a connection, a set, or a property of the whole corpus)? If eight are lookups, a graph buys you nothing, and the budget belongs in better chunking, hybrid keyword search and reranking. If six are relational, you have a case.

Two more situations where the answer is no. If a well-planned agent loop — search, read, search again with what it learned — gets the multi-hop answers you need, agentic retrieval is cheaper to build, if slower and less predictable per query. And if your relationships are already in a relational database — orders, customers, SKUs, suppliers, all with foreign keys — do not extract a graph from documents describing that data. Point a text-to-SQL layer at the database instead. Those relationships are already modelled, already clean, already current. Rebuilding them by having a model read PDFs is a downgrade dressed up as innovation.

Frequently asked questions

Do I need a dedicated graph database?

Not to start. A graph of a few hundred thousand edges sits fine in Postgres with two tables and recursive queries, which is far easier to operate than adding a database to your stack. Dedicated graph stores earn their place when traversals get deep, when path queries dominate the workload, or when you want a real query language for exploration. Choose one after the schema has stopped moving.

How reliable is LLM entity extraction on messy documents?

Good enough to be useful, not good enough to trust unattended. Expect it to be strong on clearly stated relationships, weaker on implied ones, inconsistent in how it names the same relation across documents, and occasionally willing to assert an edge the text does not support. The mitigations are dull and effective: constrain the schema to the entity and relation types you actually query, keep a source pointer on every edge, and sample-check the output on documents you know well before trusting the rest.

How do you keep the graph up to date?

Version the extraction per document. When a document changes, re-extract that document and reconcile: add new edges, retire the ones only the old version supported, leave edges corroborated elsewhere alone. Community summaries need periodic rebuilding too, since they derive from the graph rather than the documents. It is ordinary data-pipeline work, and it is most of the running cost after launch.

Is Graph RAG the same as multi-hop retrieval with an agent?

They attack the same failure from opposite ends. An agent does the hops at query time by searching repeatedly — no build phase, but more cost per question and more wandering. A graph does the hops in advance by making connections explicit — fast and predictable per query, expensive to build and maintain. Rare, varied relational questions favour agents. Frequent, high-value ones with recognisable shapes favour a graph.

Can Graph RAG work over a product catalogue?

It can, but read the previous section first: a catalogue is usually already relational, and compatibility, bundling and category links normally live in the database rather than in prose. The genuine case is the unstructured material around the catalogue — manuals, spec sheets, supplier correspondence, support history — where “which models does this replacement part fit?” is answerable only by reading across documents. Build the graph over that and join it to your catalogue data instead of duplicating it.

What would Ecarter build first?

Almost never the graph. We start with retrieval that works and an honest list of what it gets wrong, because that list is the only thing that tells you whether the failures are relational at all. If they are, we scope the graph narrowly — a handful of entity types, the relations that matter, one document set — and run it alongside the vector index rather than instead of it. That sits within our AI development and LLM development work, behind a storefront, an internal tool or an API.

Not sure whether your hardest questions are relational or just badly retrieved? Talk to Ecarter — we will look at the actual questions before recommending anything as expensive as a knowledge graph.