Building a RAG Pipeline Over a Financial Corpus
Retrieval-augmented generation lets a model answer from documents it can actually cite instead of from memory — the hard part is retrieval quality, not the generation step everyone focuses on.
Prerequisites: Retrieval-Augmented Generation, Large Language Models
Ask a general-purpose LLM "what did this company's CFO say about margins last quarter" and it will answer from whatever it happened to absorb during training — which is stale, unverifiable, and may not even be about the right quarter. A retrieval-augmented generation, or RAG, pipeline instead searches a corpus you control, hands the model the actual relevant passages, and asks it to answer using only those. The model stops guessing and starts reading.
Over a financial corpus — filings, transcripts, broker notes, news — this matters more than in most domains, because the documents are timestamped, versioned, and often say contradictory things about the same company at different points, and being right about which document said what, and when, is often the entire point.
RAG is a search problem wearing a generation costume. If retrieval returns the wrong passage, no amount of prompt engineering on the generation step recovers the right answer — the model can only be as good as what it was shown.
The pipeline
Documents are split into chunks — a paragraph or a few hundred tokens — and each chunk is turned into a vector embedding, a numeric fingerprint positioned so that semantically similar text lands nearby in the vector space. A query is embedded the same way, and the chunks whose vectors sit closest to the query's vector are retrieved and placed in the model's prompt alongside the question. The model then generates an answer, ideally citing which chunk it drew from.
Worked example: why chunking choices change the answer
A 10-K's risk-factors section runs 4,000 words as one continuous block. Chunked naively at a fixed 200 tokens with no overlap, the sentence "the Company does not believe currency fluctuation presents a material risk" gets split so that "the Company does not believe" ends one chunk and "currency fluctuation presents a material risk" starts the next. A query about currency risk retrieves the second chunk — the negation is gone, and the model reports material risk that was, in the source text, explicitly denied.
Two fixes address this directly: chunk on sentence or section boundaries rather than fixed token counts, and add overlap between adjacent chunks so a boundary rarely falls mid-sentence. Neither fix touches the generation step at all — the bug and its fix live entirely in retrieval.
Where it breaks in practice
Financial corpora are full of near-duplicate documents — an amended filing, a corrected transcript, a wire story copied across outlets — and a naive index retrieves several near-identical chunks instead of a diverse set, wasting the model's limited context. Deduplicate or diversify retrieval results before generation. Corpora also carry a time dimension a generic RAG setup ignores: retrieving the most semantically similar chunk is not the same as retrieving the most recent one, and for anything guidance- or forecast-related, recency should usually win a tie. See Point-in-Time Data.
A model given retrieved context will still sometimes answer from its own memory instead of the passage in front of it, especially when the retrieved chunk is ambiguous or thin. Require the answer to quote the retrieved text directly, and treat any answer that can't be traced to a specific chunk as untrusted output, not as a partial win.
Related concepts
Practice in interviews
Further reading
- Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2020)