RAG in Plain English: When Retrieval Beats Fine-Tuning

A client came to us last year with 4,200 resolved support tickets and a plan: fine-tune a model on them so the bot would “know our product”. They’d budgeted six weeks and a GPU bill. We talked them out of it in one meeting, shipped a retrieval pipeline over the same tickets plus their docs in nine days, and the answers were better, because the model could cite the exact ticket it was paraphrasing.

That’s the pattern we keep seeing. Teams reach for fine-tuning when what they actually have is a lookup problem. Fine-tuning changes how a model behaves. Retrieval changes what it knows right now. Those are different problems and they get confused constantly.

So here’s the practitioner’s version: what RAG is actually doing, where the work lives, and the specific cases where fine-tuning is the right call and retrieval will waste your time.

What is RAG, minus the architecture diagram

Strip away the vendor slides and retrieval augmented generation is three steps:

1. Take the user’s question. Search your own content for passages likely to answer it.
2. Paste those passages into the prompt, with instructions to answer only from them and cite sources.
3. Send it to the model.

No training step, no weights, no GPU. If you’ve ever written a search page that queries a database and renders results, you’ve built two thirds of a RAG system. The novel part is that the “rendering” target is a language model instead of a human eye.

This matters because it reframes debugging. When a RAG answer is wrong, the question is almost never “which model should we switch to”. It’s “was the correct passage in the context window, yes or no”. Log the retrieved chunks on every request from day one. We’ve diagnosed maybe 70 percent of reported RAG bugs by reading that log and finding the answer simply wasn’t retrieved.

RAG vs fine tuning: what each one actually changes

Fine-tuning adjusts model weights on your examples. In 2026 that usually means a LoRA adapter on an open-weight model, or a managed fine-tune via an API. What it’s genuinely good at:

• Output shape. You need strict JSON in a weird schema, or every reply in exactly 40 words with your brand’s cadence. Few-shot prompting gets you to 90 percent; a fine-tune gets you to 99 and lets you drop a 2,000-token system prompt from every call.
• Classification and extraction at volume. A fine-tuned small model doing ticket triage at 30ms and a fraction of a cent per call will beat a frontier model on both cost and latency, once you have a few thousand labelled examples.
• Domain language the base model mangles. Medical coding, legal citation formats, industry shorthand.
• Behaviour you can’t express in a prompt. Refusal patterns, escalation logic, consistent persona across thousands of turns.

What fine-tuning is bad at is holding facts. Weights are a lossy, unaddressable store. You cannot update one fact, you cannot delete a customer’s data on request, and you cannot get a citation back. If your pricing page changes on Tuesday, retrieval reflects it the moment you reindex. A fine-tune needs a new training run and you still won’t trust it.

The decision rule we use on client projects: if the correct answer exists in a document somewhere, use retrieval. If the correct answer is a skill or a style, consider fine-tuning. Most business requests are the first kind dressed up as the second.

They also compose. The strongest production setups we’ve built do both: retrieval for facts, a light fine-tune or a carefully versioned prompt for format compliance. Start with retrieval, because you’ll learn what the model actually gets wrong, and that tells you whether a fine-tune is worth the money.

Chunking is the job nobody wants and everybody underestimates

Default chunking in most libraries is “split every 1,000 characters with 200 overlap”. It’s fine for blog prose. It’s actively harmful for documentation, which is the corpus people usually have.

Here’s what goes wrong. A pricing table gets split so the tier names are in chunk 7 and the numbers are in chunk 8. A code sample loses its first three lines. A heading like “Troubleshooting: connection refused” ends up in a different chunk from the fix underneath it, so the embedding for the fix has no idea what problem it solves.

Structure-aware chunking fixes most of this. Split on headings first, keep tables and code blocks atomic even if they blow the size budget, and prepend the heading path to every chunk before you embed it:

// Prepend breadcrumb context so each chunk embeds with its own topic.
// Without this, "Set the timeout to 30s" has no idea it's about webhooks.
function buildChunkText(doc, section) {
  const breadcrumb = [doc.title, ...section.headingPath].join(" > ");
  return ${breadcrumb}\n\n${section.body};
}

// Store the clean body separately: embed the enriched text,
// but show the user (and the LLM) the original.
const record = {
  id: ${doc.slug}#${section.anchor},
  embed_text: buildChunkText(doc, section),
  display_text: section.body,
  source_url: ${doc.url}#${section.anchor},
  product: doc.product,
  version: doc.version,      // metadata filters matter more than you think
  updated_at: doc.updatedAt
};

That version field is not decoration. On a docs corpus covering three major versions, filtering by version before the vector search eliminated a whole category of “the AI told me to use the old API” complaints. Metadata filters beat clever prompting every time.

One honest caveat: LLM-based “semantic chunking” approaches that score every sentence boundary are usually not worth the cost. We’ve measured the difference on a 20,000 page corpus and got a retrieval improvement of a couple of points for roughly 40 times the indexing cost. Spend that effort on reranking instead.

Hybrid search and the reranker: the two upgrades that pay

Pure vector search fails on exactly the queries technical users type: error codes, function names, SKUs, version strings. Embeddings smear ERRCONN2041 into the neighbourhood of every other error, because the whole point of an embedding is to blur surface form into meaning.

So run both and fuse. If you’re already on Postgres, you don’t need a separate vector database for anything under a few million chunks. pgvector with an HNSW index handles it, and you get transactional consistency between your content and your embeddings for free.

CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE chunks (
  id          text PRIMARY KEY,
  doc_id      text NOT NULL,
  display_text text NOT NULL,
  source_url  text NOT NULL,
  product     text,
  version     text,
  embedding   vector(1536),
  tsv         tsvector GENERATED ALWAYS AS (totsvector('english', displaytext)) STORED
);

-- HNSW: slower to build, far faster and more accurate at query time than IVFFlat
CREATE INDEX ON chunks USING hnsw (embedding vectorcosineops);
CREATE INDEX ON chunks USING gin (tsv);
CREATE INDEX ON chunks (product, version);

Then fuse the two result sets with reciprocal rank fusion. It needs no tuning, no score normalisation, and it works:

// RRF: rank-based, so you never have to normalise cosine scores against BM25 scores.
// k=60 is the value from the original TREC work and we've never needed to change it.
function reciprocalRankFusion(resultLists, k = 60) {
  const scores = new Map();
  for (const list of resultLists) {
    list.forEach((item, i) => {
      const prev = scores.get(item.id) ?? { item, score: 0 };
      prev.score += 1 / (k + i + 1);
      scores.set(item.id, prev);
    });
  }
  return [...scores.values()].sort((a, b) => b.score - a.score).map(x => x.item);
}

Take the fused top 50 and run a cross-encoder reranker over them. A bi-encoder embedding compares two vectors computed independently; a cross-encoder reads the query and the passage together, which is far more accurate and far too slow to run over your whole corpus. Over 50 candidates it’s cheap. Budget roughly 200 to 400ms for a hosted reranker, less if you self-host something like bge-reranker-v2-m3 on a modest GPU.

Then pass the top 5 to 8 to the model. Not the top 20. More context is not better context, and every passage you add is tokens you pay for on every single request. The arithmetic of tokens and cost is worth internalising before you design the prompt, because that’s where RAG budgets quietly explode.

Where RAG falls over, and what to do instead

Retrieval has sharp edges. These are the ones that have bitten us.

Aggregate and global questions

“How many open tickets mention billing?” cannot be answered by retrieving five tickets. Top-k retrieval is fundamentally local. If the question requires counting or summarising an entire corpus, you need a query over structured data, not a vector search. Route those to text-to-SQL or precomputed aggregates, and detect them with a cheap classifier up front.

Multi-hop reasoning

“Which of our enterprise customers are on the deprecated auth flow?” needs two lookups chained together. Single-shot retrieval gets one. You need an agentic loop that can search, read, and search again, which triples your latency and your cost. Sometimes worth it. Often the honest answer is to build a proper report.

Tone and compliance

If the complaint is “it doesn’t sound like us” or “it keeps breaking the output schema”, retrieval has nothing to offer. That’s fine-tuning territory, or at minimum a structured output constraint.

Tiny or adversarial corpora

Under about 50 documents, just put them all in the prompt and skip the pipeline. Retrieval adds moving parts you don’t need. We’ve deleted entire vector stores from client projects for this reason.

Measure retrieval separately from generation

The most expensive mistake is evaluating the system end to end and then guessing which half is broken. Split it.

Build a set of 50 to 100 real questions with the ID of the chunk that should answer each one. Ten minutes of a support lead’s time gets you most of the way. Then track two numbers: recall@k (how often the correct chunk appears in your top k) and faithfulness (does the final answer stay inside the retrieved text). If recall@10 is 0.6, no model upgrade will save you. Fix retrieval.

We hold the generation side to a short rubric scored by a second model and spot-checked by a human weekly, which is the same lightweight quality harness we use on any AI feature. It takes an afternoon to build and it turns arguments into numbers.

Also log, for every request: the query, the retrieved chunk IDs, the reranker scores, the final answer, and whether the user thumbed it down. That log is your dataset for everything that follows, including any eventual fine-tune.

“Just use a million-token context window”

You’ll hear this every time a new model ships. It’s wrong for three reasons, none of them theoretical.

Cost: at scale, sending 400k tokens on every request versus 4k is a difference of roughly two orders of magnitude on your bill. Latency: time to first token grows with input length, and a 300k-token prefill takes seconds, not milliseconds. Accuracy: models still attend unevenly across very long inputs, and a fact buried at 60 percent depth gets missed in ways a fact at the top does not. Prompt caching helps with the first two when your context is genuinely static. Most corpora aren’t.

Long context is a brilliant complement to retrieval. Retrieve 30 chunks instead of 5, give the model room to reason, stop worrying about truncation. It’s a poor replacement for knowing which 30 to retrieve in the first place.