Alert Fatigue and Escalation Policy
Why a live-monitoring system with too many alerts trains its own operators to ignore it — and the escalation rules that keep a genuinely urgent alert from getting lost in the noise.
A live trading system generates alerts constantly — a P&L threshold breach, a data feed lagging, a position limit near its cap. If every one of these fires the same loud notification, the humans monitoring the desk learn, quickly and rationally, to skim and dismiss alerts rather than investigate each one — this is alert fatigue, and it's dangerous precisely because it makes the one alert that actually matters just as easy to ignore as the hundred that didn't.
The standard fix is tiered escalation: alerts are ranked by severity, and only the truly urgent ones interrupt someone immediately (a phone call or a page), while lower-severity issues land in a dashboard or log to be reviewed on a normal schedule. An escalation policy also defines what happens if the first responder doesn't acknowledge an urgent alert within a set window — for example, page a backup on-call person after five minutes of silence — so a single missed notification doesn't mean the issue goes unhandled.
The design principle underneath both pieces is the same: an alerting system's real job is not to report every anomaly, but to make sure the recipient's attention is proportional to actual severity, since an alert that fires too often for too little reason is worse than no alert at all — it actively degrades response to the alerts that matter.
An alert system that treats every anomaly as equally urgent trains its operators to ignore it entirely — tiered severity levels and a defined escalation path (who gets paged, and after how long if there's no acknowledgment) are what keep a genuinely critical alert from being lost in routine noise.
Further reading
- PagerDuty, Incident Response Documentation