Modular RAG: Building a Retrieval Pipeline You Can Still Change

Every technique in the advanced RAG toolkit is worth having. The problem is how they arrive: one at a time, each a small addition to a function that already worked. Hybrid search gets a mode argument. The reranker goes behind a flag because it was too slow for one client. Query rewriting is skipped for short queries because someone measured it and it hurt.

Eighteen months later that function is 600 lines, four of its flags are load-bearing, and whoever wrote the reranking branch has moved on. Changing the reranker breaks the evaluation harness. A regression appears on Monday and nobody can say which stage caused it.

There is no accuracy trick in this post. Modular RAG is an argument about the shape of the code and the seams between its parts — about what it will cost you to change one thing next year.

How does a working pipeline turn into one nobody wants to touch?

The first version is genuinely good. Eighty lines, readable in one sitting: embed the question, search, build a prompt, call the model. That is exactly why the decay is hard to spot — every step away from it is small and defensible. The end state has recognisable symptoms:

• The evaluation harness calls the whole pipeline. It asserts on final answer strings, so a reranker swap breaks tests that have nothing to do with reranking.
• Regressions are bisected by commit, not by stage. Someone spends a day in the logs to discover that the chunk size changed.
• Two experiments cannot run at once, because the choices live in constants rather than in a configuration you can name and copy.
• Nothing is ever deleted. A branch fires for one tenant and is probably dead, but proving that is a half-day job, so it stays. Repeat for two years.
• One person is the pipeline, and every change queues behind whoever last understood it.

None of these are performance problems. The system may be answering perfectly well. What has been lost is the ability to improve it safely, which is what you were actually buying when you built it.

What does “modular” actually mean here?

Each stage becomes a component with a declared input and a declared output. How it works inside — which library, which model, whether it calls a vector store or a database — is nobody else’s business.

The boxes are the boring part. The unit of value is the interface. A retriever that takes a query, a filter set and a k, and returns passages each carrying text, a source identifier and a score — that contract is what makes the rest possible. Once it holds, a dense retriever, a hybrid retriever, a SQL query against your orders table and a graph traversal all become interchangeable to everything downstream. The reranker does not know which one ran.

The discipline is in refusing the shortcuts that quietly break it: a reranker that reaches into the raw session for a hint, a generator that fires one extra retrieval just for this one case, a retriever that returns a differently-shaped object on a cache hit. Each saves an hour and costs you the seam. Three of them and you have a modular diagram sitting on top of a monolith.

One caution, because this is where teams overshoot. Do not build a plugin framework on day one. Extract an interface when you write the second implementation, not the first — an abstraction designed from a single example is usually the wrong abstraction, and a wrong interface costs more than no interface.

Which questions should go down which path?

If you take one idea from this post, take routing. It is the modular pattern that pays for itself fastest, and usually the first real module a team extracts.

Four turns arrive at a storefront assistant within a minute of each other. What is your returns window on opened electronics? That lives in a policy document. Is the 42-inch model in stock in Dubai? That is a live inventory row, and it must never come from an index built last Tuesday. Where is order 88213? An API call with an authorisation check attached. Hi. That deserves no retrieval at all — searching a vector store to say hello is a small permanent tax on every conversation.

One retrieval path serves exactly one of those four well. That is the whole argument for a router.

Deciding the routes: do not invent them in a workshop. Take a few hundred real questions — tickets, chat logs, the sales inbox — and sort them by which data source answers them, not by topic. Routes follow sources, not intents. If two carefully distinguished intents both hit the same index with the same filters, they are one route, and splitting them buys nothing but a classification error waiting to happen.

Keeping the router simple: a small closed set of labels, three or four to start, produced by a lightweight classifier or one cheap model call. Four rules that hold up in production:

• Have a default, and make it the broad one. When confidence is low, fall back to general document retrieval, never to the most specialised path. Being vague is recoverable; confidently querying the wrong source is not.
• Let routes overlap. Running two and merging the context costs a little latency and saves a wrong answer. That trade is usually worth taking.
• Log every decision with the query. That log is the training set for the better router you will want later, and it is how you find the fifth route you did not know you needed.
• Give the router its own eval set, scored per route. A router that is right most of the time overall can still be wrong almost always on your smallest and most valuable route.

There is a pleasant side effect. Chitchat skips retrieval, stock lookups skip the reranker because a database row needs no reordering, and only the policy path pays for the expensive model. Modularity gets sold on maintainability and quietly delivers a cost reduction.

Why can you not improve a pipeline you only measure end to end?

Because an end-to-end score tells you that something got worse without telling you what. You are left holding a number and a shrug.

Retrieval quality and answer quality are separate measurements with separate fixes, and conflating them wastes more engineering time than any other habit in this field. If the passage containing the answer was never retrieved, no amount of prompt work will fix the answer — the model is being asked to invent. If it was retrieved and the answer is still wrong, changing your embedding model is pure motion.

The retrieval row is the cheap win, and almost nobody does it. Build a fixed list of questions paired with the passage that actually answers each one, then check whether that passage came back. Nothing needs a model to grade it, so the test is deterministic and fast enough to run on every commit. Answer quality is harder — a rubric, usually a model to apply it — but you run it with the context held fixed, so it measures generation and nothing else.

All of which is only possible if the stages are separable. That is the point: modularity makes measurement possible, and measurement makes improvement possible. That chain is the real return on the architecture.

Should the pipeline be configuration or code?

Once the modules have honest interfaces, a pipeline is just a list of which implementations to use and with what settings. Write that down declaratively — a named variant in a config file, not a branch in a function.

The payoff is not elegance. It is that a variant becomes something you can copy, name, run and compare: three configs against the same evaluation set in one command, results side by side. Nobody has to ask which reranker was switched on when last month’s numbers were taken, because the config that produced them is stored next to them.

Two warnings. Configuration that grows conditionals is code with worse tooling and no type checking — the moment you want an if inside your YAML, write a module instead. And config edits need review like any other change; “it was only a config change” has broken more production systems than any refactor.

When is modular RAG over-engineering?

Often. This is the part of the topic I will happily argue against, because the failure mode is real and expensive.

One corpus, one kind of question, one maintainer: a module registry is strictly more code for identical behaviour, and the interfaces will be wrong, because you are generalising from a sample of one. Write the eighty-line function. Keep it readable. The naive pipeline is not a shameful starting point — it is the correct one.

The single thing that is never premature is the evaluation set. Fifty real questions with known answers, checked in beside the code, from week one. It costs a morning, and it is what will tell you when the rest of this becomes necessary. Pay for the boundaries when any two of these are true:

• More than one data source — documents plus a database, or two indexes with different freshness guarantees.
• More than one type of question, needing genuinely different handling rather than different wording.
• More than one maintainer, including one person across a gap long enough that they read their own code as a stranger.
• You are writing an if around a retrieval step for the third time, or you want to run two retrievers against the same traffic and compare.

The cost is design effort, not runtime: a clean interface adds nothing per query. That asymmetry is why extracting a boundary at the second implementation beats anticipating one that never arrives.

It also decides how easily you climb the rest of the RAG ladder. When questions start needing facts that are connected rather than co-located — which supplier ships which component to which region — Graph RAG arrives as a new retriever behind an interface you already have. When a question needs several dependent steps chosen at runtime, Agentic RAG is structurally a router that is allowed to loop and call tools. Both are far cheaper to add to a modular pipeline than to a 600-line function.

Frequently asked questions

Do I need a framework like LangChain or LlamaIndex for this?

No. Frameworks hand you ready-made boxes and a lot of adapters, which saves real time on integrations, but modularity is fundamentally an agreement about types and responsibilities — plain functions and a couple of dataclasses will get you there. The trade-off is that a framework imposes its interfaces on you, which is comfortable until your requirements and its assumptions diverge. If you adopt one, keep your own contract at the boundary.

How many routes should we start with?

Three or four: documents, live data, and no retrieval covers a surprising amount. Add a route when your logs show a cluster of failures that a new path would actually fix, not because a category sounds distinct on a whiteboard. Every route is one more classification decision that can go wrong, so each has to earn its keep.

Does routing make the system slower or more expensive?

The classification step adds a small fixed cost to every turn, and in most systems it repays that immediately, because entire categories of traffic stop doing expensive work they never needed. It depends on your traffic mix, though — if nearly every question is the same kind of question, routing is pure overhead and you should not have it.

What happens when the router picks the wrong path?

Design for it, because it will. Fall back to the broadest route when confidence is low, allow two routes to run and merge their context, and make sure the generation prompt is willing to say it does not know rather than answer from the wrong source. A wrong route that hedges is a minor incident. A wrong route that answers fluently from an unrelated document is the one that reaches your customers.

Can we retrofit this into a pipeline we already have?

Yes, and incrementally is the only sane way to do it. Pull out retrieval first: it has the clearest boundary and the best evaluation story, so you get a measurable win early. Add a router when the second data source appears. Move the constants into config once two variants exist. Each step leaves the system working, which matters when it is already serving customers.

Can Ecarter design or refactor a pipeline like this?

Yes — architecture and refactoring of existing RAG systems is part of our AI development and LLM development work, whether the system sits inside a CS-Cart, Magento or Shopify store or behind your own APIs. We usually start by building the evaluation set and drawing the real boundaries in what already exists, before anyone writes new code.

Sitting on a RAG pipeline that works but has become risky to change? Talk to Ecarter about an architecture review — routes, interfaces and an evaluation harness, scoped to the system you already run.