Naive RAG Explained: The Baseline Pipeline and Where It Breaks

A support assistant goes live on a Monday. By Wednesday somebody drops a screenshot into the team channel: a customer asked whether order 88-2140 could still be cancelled, and the assistant answered with the returns policy instead. Fluently. Quoting a fourteen-day window that applies to something else.

Nothing crashed. No alert fired, and the logs show a perfectly healthy request. The retriever did exactly what it was built to do: it found the passages sitting nearest the question in vector space and handed them over. The model did its job too, which is to write a good answer from whatever it was given.

That is naive RAG. It is worth understanding precisely, because it is both the version you should build first and the version that produces the screenshot above.

What does “naive” actually mean here?

“Naive” is a description, not an insult. It is the standard name for the baseline configuration of retrieval-augmented generation — the plain linear chain with nothing added on top.

Offline you do three things: split the documents into chunks, run each chunk through an embedding model to get a vector, store the vectors in an index. For most corpora that is a script you run in an afternoon.

At query time you do three more. Embed the user’s question with the same model. Retrieve the k chunks whose vectors are closest to it, usually by cosine similarity. Paste them into a prompt above the question and ask the model to answer using them.

What makes it naive is everything absent. The question is not rewritten or expanded. Nothing reranks the results. No router decides which index to search, or whether to search at all. There is no second pass — whatever comes back first is what the model gets. Six steps, one direction, no loops.

Why should you build the naive version first?

Because you cannot tell which sophistication you need until something has failed in a specific way, and naive RAG is the cheapest instrument for finding out.

Plenty of teams skip it. They read about reranking, query decomposition and agentic retrieval loops, and build all of it into version one. Six weeks later they have nine moving parts, no baseline, and no way to attribute a bad answer to any single stage. Removing the reranker to check whether it was ever helping means untangling it from three components built on the assumption that it exists.

The naive build earns its keep four ways. It is quick, so you get real questions from real users early instead of imagined ones. It is legible — six steps, each printable to a log. It is cheap per query. And it sets the number everything else is measured against: if a reranker cannot be shown to move that number, it has added latency and cost for nothing.

Where does naive RAG actually break?

These recur across nearly every project. Naming them matters, because a name turns a vague “the bot is bad” complaint into a ticket someone can close.

• Bad chunk boundaries. A cancellation policy split so the condition sits in chunk 4 and the exception in chunk 5. A spec table severed from the header row naming the product. Chunking decides what it is possible to retrieve; no clever search recovers a fact cut in half at index time.
• Vocabulary mismatch. The customer writes “it arrived smashed”; your policy says “damage in transit”. Embeddings handle a lot of this — that is their whole point — but unevenly, and they are worst exactly where your internal jargon diverges most from customer language.
• Exact identifiers. SKUs, part numbers, model codes, order references. Dense embeddings capture meaning, and a part number has almost none; two codes differing by one character land close together while the one you wanted sits elsewhere. This is the most common reason a catalogue assistant looks broken to the people who use it daily.
• Top-k is a blunt instrument. k is a fixed number chosen once and applied to every question. Set it low and the harder answers never reach the prompt. Set it high and the right passage arrives surrounded by nine plausible distractors.
• Lost in the middle. Retrieval succeeded. The correct passage is in the context window. It is at position seven of ten, in the region models attend to least, and the answer comes back hedged or wrong. This one is maddening precisely because your retrieval metrics look fine.
• No concept of “nothing found”. A similarity search always returns k results. Ask about something your documents have never covered and you still get k chunks — the least-irrelevant ones — and a model instructed to answer from context will duly answer from noise. That is the confident wrong answer in the opening story.
• Multi-hop and aggregate questions. “Which suppliers ship to both the UK and the UAE?” “How many products in the winter range are out of stock?” No single chunk holds those answers, so no single retrieval can find them. They need a graph over your entities or a query planner — a structural limit, not a tuning problem.

How do you tell which failure you have?

This is the section that saves the most time, and it comes down to one question asked of every bad answer: did the system retrieve the right passage and then answer badly, or did it retrieve the wrong passage?

Those are different bugs with different fixes, and almost everybody guesses instead of checking. Guessing means fiddling with the prompt, because the prompt is the part you can see. If the problem is retrieval, the prompt is the one place a fix cannot live.

Checking is not sophisticated. Log the retrieved chunks for every request — the actual text, the similarity score, the source document and position. When a bad answer surfaces, open the log and read those chunks next to the question, before you look at the answer at all. Ask one thing: if a competent new hire were handed only these passages, could they have answered correctly?

If yes, retrieval worked and the generation step is at fault. If no, stop reading the answer entirely. It was never going to be right.

That single habit — read the context before the output — separates a system that improves every week from one where people take turns rewriting the system prompt. Build the logging on day one. A small internal page showing question, retrieved chunks and answer side by side is enough, and it takes an afternoon.

Keep the failures in a file as you find them. Question, expected answer, what came back. Thirty of those is a usable evaluation set, and it converts every later change from an opinion into a measurement.

When is naive RAG genuinely enough?

More often than the current discourse suggests. The conditions are reasonably specific, and if your project meets most of them you can stop building.

The corpus is small to moderate and written clearly, with real headings and self-contained sections. It is stable — a policy set, a product manual, an onboarding handbook — not something that changes hourly. The question space is narrow, phrased in roughly the language the documents use. Answers live in one place, so nothing requires stitching two documents together. And the cost of an occasional wrong answer is low, because a human is in the loop or the assistant is internal.

An internal HR handbook assistant hits nearly all of those. A storefront assistant covering 50,000 SKUs, live stock and part numbers hits almost none — useful to know before you write any code.

What should you fix first when it is not enough?

In roughly this order, cheapest and most diagnostic first. Resist the urge to start at the bottom because it is more interesting.

The first five are still naive RAG with the rough edges filed off. The sixth is where you cross into Advanced RAG, the next post in this series, which covers rewriting, reranking and hybrid retrieval properly. If your failures are structural instead — questions that span entities, or that need a plan before a search — the honest next step is agentic retrieval, and no amount of tuning the baseline gets you there.

Frequently asked questions

Is “naive RAG” a real term or just a way of calling something bad?

It is a real term, and descriptive rather than pejorative. It names the baseline retrieve-then-generate pipeline, in contrast to advanced and modular variants that add stages around it. Plenty of production systems serving real users are naive RAG with good chunking, and there is nothing embarrassing about that.

How many chunks should I retrieve?

Start around three to five and change it only in response to logged failures. If answers miss a caveat, the right passage is probably falling just outside k and raising it helps. If answers are confidently built on the wrong passage, raising k makes things worse — you are adding distractors. There is no correct k independent of your corpus, which is why the evaluation set matters more than the number.

What chunk size should I use?

Follow the document’s own structure before reaching for a character count. A policy section, a FAQ entry, a product record and a spec table are natural units, and splitting on them beats splitting every 500 characters. Where a fixed size is unavoidable, add overlap so a fact straddling a boundary appears in both chunks, and carry the heading into the chunk text so a retrieved table still says what it describes.

Why does my assistant get product codes and order numbers wrong?

Because dense embeddings encode meaning, and an identifier carries very little. Two codes differing by a single character can sit close together in vector space while the one you wanted sits far away. The fix is not a better embedding model; it is adding a keyword index or an exact metadata filter alongside the vector search, so identifiers are matched literally and prose is matched semantically.

Do I need an evaluation set before I start improving things?

You need one before your second change, at the latest. Without it, every improvement is a story someone tells in a meeting. Thirty real questions with expected answers, re-run after each change, is enough to catch the common case where a fix for one failure quietly breaks something that used to work.

Can Ecarter build and then improve this for us?

Yes. RAG systems, catalogue-aware search and internal assistants are part of our AI development and LLM development work, including customer-facing chatbots on CS-Cart, Magento and Shopify stores. We usually ship the naive pipeline with retrieval logging first, then add stages against the failures it actually produces rather than the ones we assumed it would.

Have a document set or catalogue you want an assistant to answer from? Talk to Ecarter about a naive RAG pilot on one narrow corpus — the fastest honest way to find out how much RAG you actually need.