Chunking Strategies for Long Filings

An annual report can run past 200 pages, but an embedding model turns text into a single vector meant to represent one coherent idea, and a language model can only read so many tokens at once. You can't hand either of them the whole filing and hope for the best — a single vector for 200 pages of prose is too blurry to retrieve anything specific, and stuffing the whole document into a prompt is slow, expensive, and often impossible. So before anything else happens in a retrieval pipeline, the filing has to be cut into pieces. How you cut it decides how well everything downstream works.

The trade-off in plain terms

Cut too small — one sentence per chunk — and each piece loses context: "this increased 12% year over year" means nothing without knowing which line item it's about. Cut too large — a whole section — and you dilute the chunk with unrelated material, so a query about litigation risk pulls back paragraphs of boilerplate along with the one sentence that mattered, and the resulting embedding resembles neither topic well. The goal is chunks self-contained enough to make sense alone, small enough to embed one idea sharply, and consistently sized so retrieval scores stay comparable across a document.

Three common strategies

Fixed-size chunking splits the raw text every $N$ tokens (commonly 200–500), usually with a small overlap (e.g. 50 tokens) between consecutive chunks so a fact split across the boundary still appears whole in at least one chunk. It's simple and fast, but blind to structure — it will cut a sentence, or a table row, in half exactly as readily as it cuts between paragraphs.

Structure-aware chunking uses the filing's own layout — section headers, item numbers (Item 1A: Risk Factors), paragraph breaks — as cut points, so each chunk respects a natural document boundary. This requires a parsing step first (see document parsing pipelines) but produces chunks that read as complete thoughts, which matters both for embedding quality and for showing a user a clean source snippet.

Semantic chunking measures similarity between adjacent sentences and cuts where the topic actually shifts, rather than at a fixed token count or a fixed heading. It adapts to how densely an idea is discussed — a single dense paragraph on debt covenants might become its own chunk, while several short related sentences merge into one — at the cost of an extra embedding pass just to decide where the cuts go.

Worked example: chunking a risk-factors section

Suppose "Item 1A: Risk Factors" runs 6,000 tokens across 12 risk sub-sections (interest rate, litigation, supply chain, etc.), each roughly 500 tokens. Fixed-size chunking at 300 tokens with 50-token overlap yields roughly $\lceil 6000 / 250 \rceil = 24$ chunks, many straddling two unrelated risk topics — a query about supply-chain risk might retrieve a chunk that's half supply chain and half litigation, hurting the similarity score for either. Structure-aware chunking on the sub-section headers instead yields 12 chunks, each matching one topic exactly, with no dilution. The extra header-detection step pays for itself directly in retrieval precision.

Structure-aware (by section header) cuts align with topic boundaries

What this means in practice

Most production pipelines over financial filings use structure-aware chunking as the primary cut (by Item number, header, or table boundary), then apply fixed-size splitting within any section that's still too long to embed as one piece — combining both rather than picking one exclusively. Chunk size is also a retrieval-quality knob worth tuning empirically: too small hurts recall of nuanced facts, too large hurts precision, and the right number tends to differ between a dense risk-factors section and a sparse cover page.