Where RAG quality is actually lost

A retrieval feature starts giving worse answers, and the first move is usually to replace the vector database. Occasionally that is the right move. More often the quality was lost earlier in the pipeline, in a stage nobody instrumented, and swapping the database reshuffles the same failures.

This piece is a way to find the stage. Every layer below can be measured on its own, which is what turns a vague complaint into a fixable defect.

Ingestion

Real document estates are hostile. Scanned pages, multi-column layouts, tables that span pages, headers repeated on every page, five versions of the same policy with different dates. If the parser mangles any of this, nothing downstream can recover it.

The tell is structural: answers that invent table contents, citations pointing at pages that do not exist, or documents the system insists are missing even though they are plainly in the index.

Chunking

Chunking quietly decides what the system can express. Chunks that are too small arrive without the context needed to answer; chunks that are too large bury the relevant passage in filler. Naive splitting also cuts straight through tables, lists and cross-references, after which the model reasons about half a table as if it were whole.

No default works across document types. Chunk size and overlap are design decisions, and they belong to the documents and the questions, in that order.

Retrieval

If the right passage never reaches the model, nothing else matters. Three causes cover most of it:

• The embedding model was chosen for generic similarity and does not know the domain vocabulary.
• Exact identifiers, codes and names need lexical matching alongside the vector search, and nobody added it.
• There is no reranker, so top-k embedding similarity is treated as a final ranking when it is a first draft.

Query understanding

A user asks about "the refund window". The document says "cancellation rights within fourteen days". Vocabulary mismatch alone can sink retrieval even when every other stage is correct, which is why query rewriting or expansion earns its place in serious systems.

Generation and grounding

Given the right context, a model can still ignore it, blend in outside knowledge, or answer without the citation that would make the answer auditable. Grounding is a design requirement: the system must be built to insist on evidence, and the evidence must be verified. It does not happen by prompting harder.

Measuring stage by stage

Build a small known-answer set: fifty to a hundred real questions, each with the document and passage that should support the answer. Then ask one question per stage:

| Stage | Question | Failure signal | | --- | --- | --- | | Ingestion | Does parsed text preserve structure? | Tables mangled in the index | | Retrieval | Is the supporting passage in top-k? | Passage in corpus, absent in context | | Rerank | Does the best passage rank first? | Right passage buried mid-list | | Generation | Does the answer follow from the context? | Right context, wrong answer | | Grounding | Is the citation correct? | Plausible answer, wrong source |

Run this over your misses and the result is a ranked engineering backlog rather than a debate. "The chatbot is dumb" becomes a concrete list: recall too low on legal questions, reranker missing on tables, generation fine. Those are things you can go fix.

Where to start

In this order: fix ingestion for the formats you actually have, choose chunking against the questions you actually get, measure retrieval recall before touching the model, add reranking and hybrid search where the numbers justify them, and hold the whole pipeline behind a regression suite wired to corpus changes.

If you only do one of those this week, build the known-answer set and measure recall. It is an afternoon of work, and it usually reorders the entire roadmap.