Retrieval-Augmented Generation

Ask a language model what a company guided to last quarter and you will get an answer. You will get an answer whether or not the model has ever seen that filing, because a next-token predictor's job is to produce plausible text, not to abstain. It has no mechanism for saying "I don't know", and the fabricated version reads exactly like the real one.

The fix is embarrassingly simple and it is the difference between a demo and a system: stop asking from memory, and hand over the document. A closed-book exam rewards recall and punishes anyone who studied a different chapter. An open-book exam rewards finding the right page. Retrieval-augmented generation, or RAG, turns every question into an open-book exam — search your own corpus for the passages that matter, staple them into the prompt, and require the answer to come from them.

The pipeline

There are two halves, one that runs ahead of time and one that runs per question.

Offline, you take every document — filings, transcripts, research notes, internal wikis — and cut them into chunks of a few hundred tokens. Each chunk is turned into an embedding vector by a model that places semantically similar text nearby, the same geometry described in Word Embeddings and Word2Vec. Those vectors go into an index built for fast nearest-neighbour lookup, described in Vector Databases and Embedding Search.

Online, the user's question is embedded with the same model, the index returns the $k$ closest chunks, and those chunks are pasted into the prompt above the question with an instruction like "answer only from the passages below, and quote the sentence you used." The model generates, and because the source text is right there, every claim can be checked against it.

Worked example: ranking the chunks

Retrieval is cosine similarity, nothing more. Suppose a question embeds to $q = (0.6,\ 0.8,\ 0)$, which has length $1$, and three candidate chunks embed to:

• $c_1 = (0.5,\ 0.7,\ 0.1)$ — a paragraph on margin guidance
• $c_2 = (0.9,\ 0.1,\ 0.4)$ — a paragraph on share buybacks
• $c_3 = (0.0,\ 0.2,\ 0.98)$ — a boilerplate legal disclaimer

For $c_1$: the dot product is $0.6(0.5) + 0.8(0.7) + 0(0.1) = 0.86$ and $\lVert c_1 \rVert = \sqrt{0.75} = 0.866$, so

$$\cos(q, c_1) = \frac{0.86}{0.866} = 0.993 .$$

For $c_2$: dot product $0.62$, length $\sqrt{0.98} = 0.990$, giving $\cos = 0.626$. For $c_3$: dot product $0.16$, length $1.000$, giving $\cos = 0.160$.

Ranking is $c_1$, then $c_2$, then $c_3$. With $k = 2$ the first two chunks go into the prompt and the disclaimer is correctly left out. In plain English: the index is voting on which paragraphs the question is about, using nothing but angles between vectors.

Worked example: where the accuracy goes

RAG accuracy is a product of two independent stages, and knowing which one is failing is most of the job.

Say your retriever puts the correct passage somewhere in the top $k$ for $85$ out of $100$ questions — recall at $k$ of $0.85$. And say that when the right passage is present, the model extracts the answer correctly $95$ times out of $100$. The end-to-end ceiling is:

$$0.85 \times 0.95 = 0.8075$$

About one answer in five is wrong, and three-quarters of that error is retrieval's fault. Buying a larger or cleverer language model addresses the $0.95$ and can win you at most a couple of points. Fixing chunking, adding keyword search alongside the embeddings, or re-ranking the top 50 down to the top 5 addresses the $0.85$, where the real damage is.

The other constraint is arithmetic. With an $8{,}000$-token context, reserving $1{,}000$ for the answer and $300$ for instructions leaves $6{,}700$ for retrieved text. At $600$ tokens per chunk that is $k = 11$ chunks, and no more. Bigger chunks mean fewer of them; smaller chunks mean more of them but each one may be cut mid-argument. A common compromise is to retrieve small chunks for precise ranking, then expand each hit to the surrounding paragraph before it goes into the prompt, so the model sees enough context to interpret what it matched on.

Two cheap upgrades usually pay for themselves before anything exotic. Running a keyword search alongside the embedding search catches exact matches — tickers, section headings, defined terms — that vector similarity blurs away. And re-ranking a generous candidate set down to a handful with a slower, more accurate model shifts effort to the stage where the accuracy is actually leaking.