Agentic RAG: When Your Retrieval Pipeline Starts Making Decisions

A customer writes to your support assistant: “You told me in March that the replacement filter for the KX-200 was covered under warranty. Is it still, and can you get one to Pune this week?”

Answering that honestly needs three systems: the customer’s past correspondence, the current warranty terms for that product line, and live stock plus a delivery estimate for that pincode. The third lookup only makes sense once you know which filter part number the KX-200 takes — which you learn from the second. No single similarity search gets there. Not with better chunking, not with a reranker, not with a bigger context window.

This is where agentic RAG comes in: the point on the RAG maturity ladder where the pipeline stops being a pipeline.

What makes RAG “agentic”?

In naive RAG, advanced RAG and even modular RAG, retrieval is something the system does on a fixed schedule. Always once. Always the same way. You wrote the sequence — embed, search, rerank, generate — and it runs identically for “what is your return window” and for the KX-200 question. The model is the last stage in a machine you built.

Agentic RAG moves the control flow out of your code and into the model. You give it a goal and a set of tools; it decides the order of operations. That is the whole shift, and everything else in this post follows from it.

Concretely, here is what stops being your decision:

• Whether to retrieve at all. Underrated, and the cheapest win of the lot. “Summarise what I just pasted” needs no search. A fixed pipeline runs one anyway and sometimes makes the answer worse.
• Which source to use. Policy question → document index. Stock question → product database. Question about this order → order service.
• How to phrase the query. Not a rewrite rule you wrote in advance, but a query composed for this step, often using something learned in the previous one.
• Whether the evidence is sufficient — whether it can actually answer, or is about to guess.
• Whether to retry, reformulate or stop. Same source with a better query, a different source, or enough — write the answer.

You are trading a system you can reason about for a system that can reason. Most write-ups only mention one side of that trade.

What does the loop actually look like?

Underneath the marketing, agentic RAG is a short cycle repeated until an exit condition fires: plan → act → observe → critique → repeat or answer.

The blue path back into the agent is the entire difference between this and every earlier type. Graph RAG changes what you search. Modular RAG changes how you assemble the search. Agentic RAG changes how many times you search, and who decides.

One honest caveat about the critique step, because it is routinely oversold. A model grading its own retrieval is useful — it catches empty result sets, off-topic passages, and the case where the question asked for two things and only one was found. But it is the same model, with the same blind spots, that chose the query in the first place, and it will sometimes declare thin evidence sufficient and stop, confidently. Self-critique lowers the rate of unsupported answers. It does not measure quality, and it is no substitute for an evaluation set you run on every change.

Where does agentic RAG earn its keep?

Not on single-fact lookups. It earns its keep when the answer must be assembled from sources that do not live together — a document index, a live database, a search API, an internal service behind an API. “What did we promise this customer, and is that item in stock?” is two questions wearing one coat, answered by two different systems.

Take the KX-200 question from the top and watch the loop run:

1. Plan. Three unknowns: what was promised, what the warranty covers now, whether the part can reach Pune this week. Start with the promise — the rest depends on which product and which part.
2. Act → observe. Calls the correspondence and order service. Gets the March thread and the order: a KX-200 bought 14 months ago.
3. Critique. Incomplete — the thread says “covered” without stating a period. Not enough. Loop.
4. Act → observe. Queries the document index, and note the query it can now write: warranty terms for KX-200 consumables, not the vague thing the customer typed. Gets a clause — filters covered for 12 months, the unit for 24.
5. Critique. That contradicts the March message. Enough for the warranty half, and it has surfaced a real conflict rather than averaging it away. Still needs stock. Loop.
6. Act → observe. Queries the product database for the filter part number from the spec, plus a delivery estimate for the pincode.
7. Answer. Filter cover expired at 12 months; the March reply was about the unit, not the consumable; the part is in stock and can ship — and given what we said in March, a goodwill exception is a call for a human.

Four model calls, three tools, and a query at step four that could not have been written before step two finished. That dependency is the tell. If the second query needs the first answer, no amount of tuning a one-shot retriever gets you there. It is also the honest boundary between a RAG pipeline and agent development — retrieval is now one tool among several.

What does it cost you?

Bluntly: every iteration is another model call. Four turns is four round trips of latency, four sets of input and output tokens, four chances for something to go sideways. Against a fixed pipeline, agentic RAG is slower and more expensive per query — and neither is a number you can put in a spreadsheet.

That last part is the one people underestimate. Agentic RAG is non-deterministic. The same question, asked twice, can take a different path — different tool order, different number of iterations, a different final answer. Not usually a wrong one. Just a different one. If your test suite asserts on exact output, it will flap.

For a storefront chat widget where a shopper expects an answer before they lose interest, several sequential model calls is a product decision, not just an engineering one. Usually the right design is a fast fixed pipeline for the common questions and an agentic path only where it is clearly needed — with a visible “checking a few things” state so the wait is explained rather than merely endured.

What guardrails do you have to build?

These are not polish. Ship agentic RAG without them and you will eventually meet a query that loops until something times out and someone notices the bill.

• A hard iteration cap, enforced in your code, not requested in the prompt. When it trips, the system stops and reports what it found so far.
• A per-query budget. Count tokens and tool calls; abort when it is spent. One pathological question should not cost what a thousand normal ones do.
• Timeouts at every level — per tool call and for the whole request. A slow internal service should degrade the answer, not hang the conversation.
• A deterministic fallback path. When the loop fails, times out or hits the cap, fall through to plain retrieval-and-answer — the boring pipeline you already trust. This one decision is what makes the feature safe to leave switched on.
• The full trace, logged. Every decision, every query it composed, every result, every critique verdict. You cannot reproduce a failure in a non-deterministic system, so the trace is the reproduction. Log it from day one; retrofitting it after a bad week is miserable.

One more thing, not strictly a guardrail: give each tool a narrow, clear description. Most bad routing we see is not the model being stupid — it is two tools described so similarly that nothing could have chosen between them.

When is agentic RAG justified — and when is it over-engineering?

The unfashionable answer: most production systems should stop at advanced or modular RAG. A well-tuned retriever with a reranker, hybrid search and honest evaluation answers the overwhelming majority of real questions, does it in a fraction of the time, and can be debugged by reading a log line. Agentic RAG on top of mediocre retrieval is just a mediocre retriever called five times.

It is justified when questions are genuinely open-ended — research-style, “compare these and tell me what changed” — or when answers must combine sources no single index can hold, especially live operational data alongside written documentation. Internal analyst tools, support across a large product portfolio, procurement and compliance research: all reasonable fits.

It is fashion when someone specifies “agentic” before anyone has written down ten questions the current system gets wrong. Read the failures. If they say the retriever found the wrong passage, you need better retrieval, not an agent. If they say no single passage could have answered this, now you are in the right room.

Frequently asked questions

Is agentic RAG the same thing as an AI agent?

They overlap, and the line is about permissions. Agentic RAG is an agent whose tools are all read-only — it searches, queries, looks things up, then answers. A general AI agent may also take actions with consequences: issuing a refund, updating a record, sending an email. Same loop shape, very different blast radius. A system that can write to your database deserves far heavier review than one that can only read.

How many loop iterations should I allow?

Start low — a small single-digit cap — and raise it only if your traces show real questions hitting the limit while genuine progress is still being made. Long loops are usually not thinking harder; they are stuck, rephrasing the same failing query in slightly different words. A low cap surfaces that fast, which is exactly what you want in the first few weeks.

Can I use agentic RAG in a customer-facing storefront chat?

You can, but be deliberate. Shoppers expect near-instant replies, and several sequential model calls are noticeably slower than one. The pattern that works is routing: answer the common questions — shipping, returns, sizing, is-it-in-stock — with a fast fixed pipeline, and reserve the agentic path for messy multi-part questions.

Does agentic RAG fix bad retrieval?

No. It can paper over a near miss with a second query, which is genuinely useful, but the ceiling is still set by what your sources contain and how well they are indexed. If the answer is not in the corpus, or chunking split it in half, the agent loops, fails, and either gives up or invents something. Fix retrieval first. Agentic RAG multiplies the quality of your tools — in both directions.

How do you evaluate a system that is non-deterministic?

Stop asserting on exact strings and start scoring outcomes. Take a fixed set of questions with known answers, run each several times, and measure how often the final answer is right, plus the spread of iterations, latency and cost. Score the steps separately too: did it pick the right tool, did it retrieve the right evidence. Variance is itself a metric — a question that answers correctly four times in five is telling you something.

Can Ecarter build an agentic RAG system for us?

Yes — multi-source retrieval, tool-calling assistants and catalogue-aware AI are part of our AI development and LLM development work, and we integrate them with CS-Cart, Magento and Shopify stores as well as standalone systems. Expect us to push back first, though: we prefer to prove a simpler pipeline against your real questions before adding a loop, because that is the version you can afford to run and debug.

Not sure whether your questions need an agent or just better retrieval? Talk to Ecarter — bring ten questions your current system gets wrong and we will tell you which of the two it is.