Calibrating Alert Thresholds and False Alarm Rates

Every monitoring system built for a trading desk faces the same tension: set the alert threshold too sensitive and it fires constantly on ordinary noise, until the people receiving it stop reading them carefully — a phenomenon sometimes called alert fatigue. Set it too loose and real problems slip through unflagged until they've already become expensive. Calibrating a threshold means deliberately choosing a point on that tradeoff, rather than picking a round number that feels intuitively "about right" and hoping it works.

The practical way to do this is to look at the threshold's behavior historically before deploying it live. Run the proposed rule against past data and count how often it would have fired, and how often those firings would have corresponded to an actual problem versus ordinary variation. A threshold that would have triggered forty times over the past year, of which only two corresponded to genuine issues, has a false alarm rate too high to be useful — by firing so often on nothing, it trains the recipient to stop responding, which defeats the purpose the two times it mattered. Conversely, a threshold that never would have fired even during a period containing a known, real incident is set too loose to do its job.

A concrete example: a desk considers an alert that fires whenever daily tracking error against the backtest exceeds 40 basis points. Backtesting this rule against eighteen months of live history shows it would have fired eleven times, and manual review finds only one of those corresponded to an actual data problem — the rest were ordinary daily noise. Loosening the threshold to 70 basis points reduces the firings to three, two of which include the same real incident plus one new borderline case worth a quick look. The looser threshold catches the problem that matters while cutting the noise enough that the alert is still being read carefully when it does go off.