System Health Metrics for Trading Systems

A strategy can be perfectly well designed and still lose money because the system running it is broken — a slow data feed, a stuck order queue, a server that silently stopped updating positions three hours ago. System health metrics monitor the plumbing itself, separately from strategy performance, because a P&L chart alone cannot distinguish "the strategy is having a normal bad day" from "the strategy stopped receiving live prices two hours ago and is trading on stale data." These are very different problems requiring very different responses, and only infrastructure-level monitoring catches the second one before it becomes a large loss.

Typical metrics include order-to-fill latency (how long between sending an order and getting a confirmation), the reject rate on orders (a sudden spike often signals a connectivity or risk-limit misconfiguration issue), data feed staleness (how many seconds since the last tick was received for each instrument), CPU and memory usage on the machines running the strategy, and heartbeat checks confirming that every process expected to be running actually is. None of these say anything about whether the trading idea is good — they say whether the system executing that idea is doing what it's supposed to.

A concrete example: a market-making strategy's P&L looks unremarkable on a given afternoon — flat, neither particularly good nor bad. System health monitoring shows that data feed staleness for one of its traded instruments spiked to 45 seconds around 2pm, well above the sub-second latency the strategy assumes when quoting. During that window the strategy was quoting off stale prices without anyone noticing from the P&L alone, because the market happened not to move much during those 45 seconds — a matter of luck rather than the system working correctly. The next time that feed hiccups during a fast market, the same latent bug could produce a real loss; system health metrics caught the underlying issue before that happened.