Multimodal RAG: When the Answer Is an Image, a Table or a Scan

A customer asks whether your jacket fits a 108cm chest. The answer exists. It is in the size chart — a PNG some merchandiser exported from a spreadsheet three years ago and dropped into the description. Your pipeline read the paragraph above it, embedded that, and returned it happily. The assistant then writes a fluent, useless answer about the jacket having a relaxed cut.

Nothing logged an error. The document was ingested, a chunk was retrieved, the model answered. Every stage reported success, and the one fact the customer needed was invisible from the first minute of indexing onward.

A framing point first: multimodal RAG does not sit on the complexity ladder described in the main RAG guide. It is orthogonal to it. You can bolt it onto a naive pipeline in a week, or onto an agentic one where a planner decides mid-loop to go look at a page image. The question is never whether you are advanced enough yet. It is what share of your answers live in something that was never a sentence.

What breaks when the answer isn’t prose?

A great deal of commercially valuable information was never written as prose. It was drawn, tabulated, photographed or scanned. Four patterns recur:

• The size chart as an image. Every measurement a customer needs, encoded as pixels. To a text extractor the page holds a heading, two marketing sentences and an image tag.
• The spec sheet where the specs are a rendered table. Extraction pulls the words out as a stream — Voltage 240 Weight 4.2 Depth 610. The numbers survive. The relationships between them do not.
• The scanned supplier invoice. No text layer at all — a photograph of a piece of paper. Without OCR it is an opaque blob with a filename.
• The garment photo, when the question is about the garment. “Does this have a mandarin collar?” takes a human a quarter of a second and is unanswerable from “classic everyday shirt, breathable cotton”.

Only the third fails loudly. The others give you a populated index, plausible retrieval and confident answers. If your RAG system tested well and then drew complaints about specific factual questions, start here.

What are the three ways to do multimodal RAG?

Three approaches are in genuine production use, and they differ enormously in cost and in what they throw away. Most teams end up running two of them for different content types — which is much easier if the pipeline is structured modularly, so a route can be swapped without disturbing the rest.

1. Caption then index. Run a vision model over every image once, at ingest, and store the description as ordinary text. Retrieval never changes — the caption is just another chunk. Cheapest thing that works, drops into an existing text stack in an afternoon, and you pay per image rather than per query. The catch is genuine: a caption is a lossy compression chosen without knowing the question. If it says “a size chart for men’s outerwear”, you have indexed that a chart exists, not what it says. Prompting the captioner to transcribe rather than summarise — every number, every label, every axis — helps a lot. You are still guessing at the questions in advance.

2. Joint embedding. Map images and text into one shared vector space, so “navy shirt with a mandarin collar” and a photograph of exactly that land near each other. No captioning step, no intermediate text, and image-to-image search comes almost free. The elegant option, and the right one for “find things that look like this”. It is weak exactly where the others are strong: fine detail and dense text inside an image. A joint model retrieves a size chart confidently, because it looks like a size chart. It cannot tell you what is in row four.

3. Vision-language model over page images. Stop parsing. Render each page as an image, retrieve whole pages, let a model read them at answer time. Layout, table structure, footnotes, stamps, handwriting in the margin — all preserved, because nothing was thrown away. On messy real-world documents this is markedly the most accurate, and also the slowest and priciest per query, since images go through the model on every request and eat context.

Start with captioning, and be aggressive about what you ask the captioner to produce. Not because it is best — it plainly is not — but because it tells you within days whether visual content is really where your failures live, without committing you to a per-query cost you may not need. Then read what it still gets wrong. If the misses are “which of these looks like that”, add joint embedding. If they are “what does row four say”, go to page images for that document class and stop trying to parse it. Routing by document type is normal. There is no prize for using one mechanism everywhere.

Why do tables need their own route?

Tables are the most common multimodal case and the most under-served.

A table’s meaning is not in its words. It is in its geometry. “610” means nothing; “610” in the Depth (mm) column of the KX-40 row means something precise. Flatten that to a text blob and you keep every token while destroying the only thing that made them answerable. The chunk still embeds nicely, still retrieves for spec questions, and hands the model a soup of numbers with no way to tell which belongs to which. The model, being a model, picks one.

Extract the structure; do not flatten it. Get the table into real rows and cells, then serialise it in a form that preserves the association. Markdown tables, HTML tables and one-sentence-per-row renderings (“For KX-40: depth 610mm, voltage 240V, weight 4.2kg”) all beat a flat blob. The last is often strongest, because each row becomes independently meaningful and independently embeddable.

Keep the header with every chunk. Unglamorous, and it matters more than almost anything else here. A 200-row size chart will not fit in one chunk; split it and rows 40 to 80 arrive with no column names, which makes them worse than useless. They are confidently wrong. Every chunk of a table should carry the header row, ideally the caption too. Same for units and footnotes — if the footnote says centimetres and sits in a different chunk, someone gets a jacket that does not fit.

When do you need OCR, and how good does it have to be?

You need OCR whenever a document has no text layer: scans, faxes, photographed paperwork, and a depressing number of supplier PDFs that turn out to be scans wearing a PDF costume. Try extracting text and see what comes back. If a 40-page catalogue yields nine words, you are looking at images of pages.

OCR sets a hard ceiling on everything downstream. No amount of clever chunking, reranking or prompting recovers information that was never read off the page. If OCR turns a part number into KX-4O with a letter O, no retriever will match a customer asking about KX-40, and no model can infer the correction.

So the priority order is the reverse of what most teams assume: fix scan quality and OCR configuration before spending anything on retrieval sophistication. Rescanning at higher resolution, deskewing pages, or switching to an engine that handles your language and layout properly beats any amount of reranking on a corpus of bad scans. And for genuinely difficult documents — carbon copies, stamped paperwork, handwritten annotations — the honest answer is often to skip OCR and let a vision model read the page image, since it resolves ambiguity with context in a way a character-level engine cannot.

Where does this pay off in e-commerce?

Three uses justify the work most often in the stores we build for.

Answering from spec sheets and size charts. The highest-value and least glamorous. Sizing and specification questions drive a large share of pre-purchase contact and of returns. An assistant that reads the actual chart, says “a 108cm chest is a size L in this cut”, and shows which chart it read, deflects tickets and prevents returns at once. It also has to be right, which is why the header rule above is not pedantry.

Visual search. “Find me more like this”, from an uploaded photo or the product already on screen. Joint embedding earns its place here, and it is one of the few AI features customers understand without explanation. Strong on visually driven categories — apparel, furniture, tiles, lighting — and close to pointless where the decision is a specification.

Extracting structured data from supplier documents. Purchase orders, invoices, packing lists and data sheets arrive as PDFs and scans, and someone in operations retypes them into the catalogue. Turning that into an extraction pipeline — page image in, validated record out, human review on anything low-confidence — is often the fastest-paying AI project in a mid-size retailer, and needs no chatbot anywhere in it. We build this as part of our AI development and e-commerce AI work, usually feeding results into the store or ERP over an API integration rather than a new interface.

How do you tell whether it actually works?

Evaluation matters more here than in any other RAG variant, for one reason: the characteristic failure is an omission, and omissions are invisible in a demo. A wrong passage you can see. A silently ignored table looks exactly like a correct answer that happens to be a bit vague. Demos are made of the questions someone thought to ask — which are the questions the captions already cover.

• Build a test set where the answer exists only in the visual element. If the surrounding prose could answer it too, the question proves nothing.
• Measure coverage at ingest, not only accuracy at query time. Count the images, tables and scanned pages in the corpus; count how many produced anything in the index. The gap is your silent failure rate, measurable before anyone asks a question.
• Check for the right source, not just the right answer. A model can produce a correct-sounding number off a plausible wrong row. If the answer carries no page or table reference, nobody can audit it.
• Read your raw OCR output by hand. An hour with a few dozen pages tells you more about your ceiling than any metric.
• Test refusal. A pipeline that invents a measurement is worse than one that says it cannot find the chart — and that behaviour degrades first when you tune for helpfulness.

Frequently asked questions

Do I need a vision model for the whole pipeline, or only at the end?

It depends which approach you pick, and the difference is financial. Captioning and joint embedding use a vision model at ingest only — once per document, then never again — so your answering model can be text-only. Retrieving page images needs a vision-capable model on every request that touches an image. Fixed one-off cost versus recurring per-query cost, which usually decides it for a high-traffic storefront.

Can I just run OCR over everything and treat it as a text problem?

For clean, text-heavy scans, yes, and you should — it is far cheaper than the alternatives. It falls apart on two things: documents where layout carries meaning, like multi-column spec tables and forms, and images with no text at all, like a product photograph. OCR gives you the characters, not the structure they sat in.

How do I stop image retrieval making every query expensive?

Route rather than blanket-apply. Most queries never need an image, so filter first and pull page images only when the question is visual or text retrieval came back weak. Retrieve fewer pages than you would text chunks, since each costs far more context. Then cache — the same size chart gets asked about hundreds of times.

Does joint embedding replace normal text search?

No, and treating it as a replacement reliably makes retrieval worse across the board. Shared-space models trade some text precision for the ability to span modalities. Keep your text retrieval as it is, run an image path alongside it, and merge results at the end — the same argument as combining semantic and keyword search. You want both signals, not one compromise signal.

What about charts and graphs, where the answer is a trend?

Harder than tables, and worth being upfront about. Reading a value off a plotted line means interpolating between axis ticks, and models do that unreliably — more so on dense or unlabelled charts. If the underlying data exists anywhere as numbers, index the numbers and treat the chart as a pointer to them. If it does not, expect approximate answers and show the chart beside the response so a human can check it.

Can Ecarter build this for us?

Yes — document extraction, OCR pipelines, catalogue-aware search and vision-enabled assistants are part of our LLM development and AI chatbot work, and we integrate them into CS-Cart, Magento and Shopify e-commerce builds as well as standalone systems. We normally start by auditing what share of your real questions need visual retrieval at all — for some catalogues it is nearly all, for others almost none.

Sitting on spec sheets, size charts or scanned supplier paperwork your search cannot see? Talk to Ecarter about a multimodal pilot on one document set — and read Agentic RAG if your questions also need more than one lookup.