Hallucination and Groundedness

Ask a large language model what a company's Q3 revenue was, and it may answer with a specific, confident-sounding number that is simply wrong — not a rounding error, not a guess flagged as uncertain, but a fabricated figure stated in exactly the same fluent tone as a correct one would be. This is called hallucination, and it isn't a rare bug to be patched away — it's a direct consequence of how these models generate text in the first place.

The analogy: a fluent talker who never says "I don't know"

Imagine a colleague who is an excellent, articulate speaker but has never once said "I'm not sure" — asked something they don't know, they answer anyway, in exactly the same confident tone as when they genuinely know the fact. A language model is like that colleague by construction: it's trained to produce the most statistically plausible continuation of text, and a fluent, specific-sounding wrong answer is often more "plausible" as a continuation than an honest admission of not knowing.

Why it happens, and how groundedness fixes it

A language model generates text by predicting likely next words given everything before it — it has no built-in mechanism separating "I recall this fact from training data" from "this is a plausible-sounding number to put here." Both processes produce fluent text the same way, so confident tone carries no information about whether a claim is true.

Groundedness is the fix: instead of asking the model to answer purely from what it "remembers," retrieve the actual relevant source text (a specific filing paragraph, a specific transcript excerpt) and include it directly in the prompt, instructing the model to answer only using that provided text and to say so explicitly if the answer isn't in it. This is the core idea behind retrieval-augmented generation. A grounded answer can be checked against its cited source; an ungrounded one can only be checked against the real world after the fact.

Worked example: ungrounded versus grounded on the same question

Ungrounded prompt: "What did Acme Corp's CFO say about margins on the Q2 call?" Without any transcript text provided, the model may generate a plausible-sounding paraphrase that blends details from other companies' calls it saw during training, confidently attributed to Acme's CFO — a hallucination.

Grounded prompt: the actual Q2 transcript excerpt is retrieved and inserted into the prompt, followed by "Using only the text above, what did the CFO say about margins? If the transcript doesn't address this, say so." Now the model's answer can be directly checked, sentence by sentence, against the provided excerpt — and if the transcript genuinely never discusses margins, a well-prompted grounded model is far more likely to say so rather than invent an answer.

Worked example: measuring a hallucination rate

A team tests a grounded question-answering pipeline on 200 filing questions where the true answer is known in advance. The model answers correctly and traceably to the source text in 178 cases, answers incorrectly but still cites a real (wrong) passage in 14 cases, and fabricates an answer with no real supporting passage in 8 cases. The hallucination rate here is $8/200 = 4\%$ for full fabrication, separate from the $14/200 = 7\%$ rate of citing genuine but irrelevant source text — two distinct failure modes that a single "accuracy" number would blur together.

What this means in practice

Any LLM-based tool used in a financial workflow needs its outputs treated as unverified until traced back to a real source, because fluency provides zero evidence of accuracy. Grounding via retrieval reduces but does not eliminate hallucination: a model can still misread a correctly retrieved passage, so it's a mitigation, not a guarantee.