Novelty and Staleness Detection in News

A company's stock drops 4% the moment news breaks that a factory has been shut down by a fire. Over the following three days, a dozen more articles cover the same event — updates on repair timelines, analyst reactions, a recap in a weekly roundup — all mentioning the fire, all scoring as strongly negative sentiment if fed naively into a sentiment aggregator. If the aggregator treats each of those twelve articles as fresh negative information, it will keep signaling "sell" for days after the market has already fully priced in the original event, and it will be wrong every time, because there's no new information in article number nine — just the same fact being retold.

What novelty detection is actually checking

Novelty detection asks, for each incoming article, whether it contains information not already reflected in recent coverage of the same entity — as opposed to staleness, which flags an article as a repeat, recap, or minor variation of something already known. The distinction matters because a price already reacted to the first mention of an event; subsequent coverage of the same fact shouldn't generate a second reaction, but a genuinely new fact — even about the same underlying event, like "repairs will take three months longer than initially estimated" — should.

Common approaches

Similarity to recent coverage: embed each new article and compare it to embeddings of recent articles about the same entity; a very high similarity score to something published in the last day or two suggests a rehash rather than new information. Named-entity and fact-delta comparison: extract the specific claims or numbers in an article (dates, dollar figures, named people or events) and compare them against a running record of what's already been reported about that entity, flagging only articles that introduce a fact not already in the record. Source and byline heuristics: wire-service syndication and republication under different bylines are extremely common for the same underlying story, so tracking whether a "new" article is really the same wire story re-published elsewhere catches a large share of pure staleness cheaply, before any content comparison is needed.

Worked example: scoring five articles about one event

Five articles about the factory fire arrive over three days. Article 1 (day 0): "Factory X shut down after fire" — the original, novel event. Article 2 (two hours later): a near-identical wire republication, flagged stale on similarity alone. Article 3 (day 1): "Analysts cut estimates following the factory fire" — high surface similarity to article 1, but a fact-delta check catches a genuinely new number (the estimate cut) that a similarity-only detector would miss. Article 4 (day 2): "Repairs will take until Q3, longer than expected" — clearly novel. Article 5 (day 3): a weekly roundup mentioning the fire among five other stories — almost entirely a recap, flagged stale. A similarity-only detector correctly catches articles 2 and 5 but would also wrongly flag article 3 as stale, since fact-delta comparison, not similarity alone, is what catches its new estimate-cut information.

What this means in practice

A sentiment or news-reaction signal that skips novelty detection tends to keep firing on stale rehashes of the same event, effectively autocorrelating with itself and adding noise rather than new predictive information — and it's also a common way for a backtest to look better than a live strategy, because backtests over historical news often understate how much of the daily article volume is genuinely repetitive.