Chunking Strategies for Long Filings

An annual report can run past 200 pages, but an embedding model turns text into a single vector meant to represent one coherent idea, and a language model can only read so many tokens at once. You can't hand either of them the whole filing and hope for the best, a single vector for 200 pages of prose is too blurry to retrieve anything specific, and stuffing the whole document into a prompt is slow, expensive, and often impossible. So before anything else happens in a retrieval pipeline, the filing has to be cut into pieces. How you cut it decides how well everything downstream works.

The trade-off in plain terms

Cut too small, one sentence per chunk, and each piece loses context: "this increased 12% year over year" means nothing without knowing which line item it's about. Cut too large, a whole section, and you dilute the chunk with unrelated material, so a query about litigation risk pulls back paragraphs of boilerplate along with the one sentence that mattered, and the resulting embedding resembles neither topic well. The goal is chunks self-contained enough to make sense alone, small enough to embed one idea sharply, and consistently sized so retrieval scores stay comparable across a document.

Three common strategies

Fixed-size chunking splits the raw text every $N$ tokens (commonly 200–500), usually with a small overlap (e.g. 50 tokens) between consecutive chunks so a fact split across the boundary still appears whole in at least one chunk. It's simple and fast, but blind to structure, it will cut a sentence, or a table row, in half exactly as readily as it cuts between paragraphs.

Structure-aware chunking uses the filing's own layout, section headers, item numbers (Item 1A: Risk Factors), paragraph breaks, as cut points, so each chunk respects a natural document boundary. This requires a parsing step first (see document parsing pipelines) but produces chunks that read as complete thoughts, which matters both for embedding quality and for showing a user a clean source snippet.

Semantic chunking measures similarity between adjacent sentences and cuts where the topic actually shifts, rather than at a fixed token count or a fixed heading. It adapts to how densely an idea is discussed, a single dense paragraph on debt covenants might become its own chunk, while several short related sentences merge into one, at the cost of an extra embedding pass just to decide where the cuts go.

Worked example: chunking a risk-factors section

Suppose "Item 1A: Risk Factors" runs 6,000 tokens across 12 risk sub-sections (interest rate, litigation, supply chain, etc.), each roughly 500 tokens. Fixed-size chunking at 300 tokens with 50-token overlap yields roughly $\lceil 6000 / 250 \rceil = 24$ chunks, many straddling two unrelated risk topics, a query about supply-chain risk might retrieve a chunk that's half supply chain and half litigation, hurting the similarity score for either. Structure-aware chunking on the sub-section headers instead yields 12 chunks, each matching one topic exactly, with no dilution. The extra header-detection step pays for itself directly in retrieval precision.

What this means in practice

Most production pipelines over financial filings use structure-aware chunking as the primary cut (by Item number, header, or table boundary), then apply fixed-size splitting within any section that's still too long to embed as one piece, combining both rather than picking one exclusively. Chunk size is also a retrieval-quality knob worth tuning empirically: too small hurts recall of nuanced facts, too large hurts precision, and the right number tends to differ between a dense risk-factors section and a sparse cover page.