Heartbeats and Staleness Detection

A trading process that has crashed and one that's simply quiet because the market is quiet look identical from the outside unless you build a way to tell them apart. A heartbeat solves this: every live component, a data feed handler, a strategy engine, a risk monitor, periodically emits a small "I'm still running" signal, independent of whether it has anything else to report. A monitoring system watches for these signals and raises an alert the moment one stops arriving, rather than waiting for someone to notice that a strategy hasn't traded in an unusually long time.

Staleness detection is the related check applied to data rather than processes: every incoming price or reference-data update is timestamped, and a monitor flags a feed as stale if too much time passes without a fresh update, a market-hours price feed that goes quiet for two minutes during active trading is almost certainly broken, not simply reflecting an illiquid market. The right staleness threshold depends entirely on context: two minutes of silence is alarming for a liquid large-cap feed during market hours but completely normal overnight or for a thinly-traded instrument.

A common design is layering both checks: the strategy process sends a heartbeat every 10 seconds regardless of market activity, while its input price feed is separately checked for staleness with a threshold tuned to that instrument's typical quote frequency, so a silent process and a silent feed trigger distinct, correctly-labeled alerts instead of one vague "something's wrong" page.