Building an Evaluation Set for Financial LLM Tasks

You build a system that extracts guidance figures from earnings calls, try it on a handful of transcripts, the answers look right, and you ship it. Two months later someone notices it's been silently wrong on a specific, common phrasing — "we expect to be in the range of" — for weeks. Eyeballing a few outputs tells you the system works on the examples you happened to look at; it tells you nothing about the examples you didn't look at, and it's exactly the systematic, recurring failure modes that a small manual check is least likely to stumble across by chance. An evaluation set fixes this by giving the pipeline a fixed, labeled body of examples with known correct answers, so performance can be measured rather than guessed at.

What a good evaluation set needs

A useful eval set has several properties a handful of ad hoc checks lacks. Coverage of edge cases, not just typical examples — ambiguous phrasing, missing data, unusual formats — because typical cases are exactly where a pipeline is least likely to fail. Ground-truth labels created independently of the system being tested, usually by a human reading the source document directly, so the label isn't circularly derived from the same process being evaluated. A fixed size large enough to detect a meaningful difference between two versions of the pipeline — testing on five examples can't reliably distinguish 80% accuracy from 95%, since the margin of error at that sample size swamps the difference. And held-out status: the eval set should never be shown to whoever is iterating on prompts or model choices during development, or improvements will implicitly be tuned to that specific set rather than to the task in general.

Worked example: sizing and building a guidance-extraction eval set

Suppose you're evaluating a pipeline that extracts a company's forward revenue guidance range from earnings-call transcripts, and you want to distinguish 85% accuracy from 95% accuracy with reasonable confidence. A rough rule of thumb for the standard error of a proportion is $\sqrt{p(1-p)/n}$; at $p=0.9$ and $n=30$, that's $\sqrt{0.9 \times 0.1 / 30} \approx 0.055$, meaning a measured 85% versus 95% could easily be noise at that sample size. At $n=200$, the standard error shrinks to about $\sqrt{0.9\times0.1/200} \approx 0.021$, tight enough to actually distinguish those two accuracy levels. So the eval set needs on the order of 150–250 labeled examples, not 20, deliberately including transcripts with no guidance given at all (to check the pipeline correctly returns "none" rather than hallucinating a number), guidance stated as a single point versus a range, and guidance withdrawn mid-call — each a distinct case the pipeline must get right, not just the easy majority case of a clean range stated once.

What this means in practice

An evaluation set earns its cost the first time it catches a regression a developer wouldn't have spotted by eye — a prompt tweak that improves outputs on the three examples someone happened to check, while quietly breaking a case elsewhere in the distribution. Once built, it should be reused every time the pipeline, prompt, or underlying model changes, turning "does this still work" from a vague impression into a specific, comparable number.