News Deduplication and Story Clustering

A single acquisition announcement can generate forty articles within an hour — the original wire release, a dozen outlets republishing it near-verbatim, several analyst commentary pieces, and a handful of aggregator sites reposting a summary. If a news-driven signal simply counts "how many articles mention this company today," it will register the acquisition as roughly forty times more newsworthy than an event that happened to generate only one write-up, when in reality there's exactly one underlying event either way. Deduplication and clustering exist to collapse that count back down to what actually matters: how many distinct stories are being covered, not how many articles happen to describe them.

Deduplication versus clustering

Deduplication identifies articles that are essentially the same text — a wire release and its verbatim or near-verbatim republications — usually via exact or near-exact text matching (hashing shingles of the text, or a very high embedding similarity threshold). This catches the cheapest, most common form of inflation: syndicated copies of one release appearing under many outlet names.

Clustering goes further, grouping articles that describe the same underlying event even when the wording differs substantially — an original wire story, an analyst's independent commentary on it, and a competitor's press release referencing the same event might use almost no overlapping sentences, but they all belong to one story cluster for the purpose of counting how many distinct events happened. This typically uses embedding similarity above a lower threshold than exact deduplication, sometimes combined with shared named entities and a tight time window, since two articles about the same company published the same day with overlapping key terms are far more likely to be one event than two.

Worked example: forty articles down to three stories

Say forty articles about a company arrive in one day. Deduplication first collapses twenty-five near-identical republications of an acquisition wire release into one representative article. Clustering then groups the remaining fifteen: eight (the wire release plus seven distinct analyst-commentary pieces on the same acquisition) form Cluster A; four discussing an unrelated executive appointment form Cluster B; the remaining three, covering a routine dividend declaration, form Cluster C. What looked like forty separate pieces of news is really three distinct events — and Cluster A, despite having the most articles, isn't necessarily the most market-moving one; it's simply the one with the most coverage, which a raw article count would conflate with importance.

What this means in practice

Any news-driven feature — sentiment aggregates, event counts, novelty scores — should be computed at the level of deduplicated story clusters, not raw articles, or it will systematically overweight whichever events happened to attract the most republication and commentary, which is a function of newsroom behavior and wire syndication as much as it is a function of the event's actual market importance. This step sits upstream of virtually every other news-analytics component and is easy to skip early on because a pipeline "works" without it — it just quietly conflates coverage volume with signal strength until someone checks.