A Monitoring Stack for Live Strategies

A strategy passes every backtest gate, ships to production, and the team moves on to the next idea. Eleven weeks later someone glances at the top-line PnL chart and it is down 4% for no obvious reason. Nobody can say when it started, because nobody was watching anything narrower than that one line. The signal had been quietly decaying since week three; the fills had been getting worse since week six; a data vendor changed a field definition in week eight and nobody noticed. Any one of those, caught early, is a one-line fix. Caught in week eleven, it is a post-mortem.

A monitoring stack is the set of instruments that would have caught it. Think of a cockpit rather than a car dashboard: a pilot does not fly on a single "how's the flight" gauge, they read altitude, airspeed, heading, and fuel separately, because a problem in any one of them is invisible in the others. A strategy needs the same separation — the PnL line is the plane's altitude, and altitude alone tells you nothing about whether the engine is failing.

The four instruments

Signal health. Is the underlying idea still working? Tracked via a rolling information coefficient or hit rate — see Monitoring Signal Decay in Production — computed on a window short enough to move before the drawdown does.

Execution quality. Are you getting the fills the backtest assumed? Fill rates, realized slippage against arrival price, queue position for passive orders — see Decomposing Live Slippage.

Risk and exposure. Are you carrying the book you think you're carrying? Gross and net exposure, sector and factor tilts, position limits, intraday drawdown — see What To Watch On The Risk Screen All Day.

Data integrity. Did an input silently change shape? Missing bars, a feed that stopped updating, a corporate action applied twice, a timestamp that jumped time zones.

Each of these is a separate time series with its own expected level and its own alert. The hard part is not building four charts — it's deciding when a wiggle in one of them is a real problem rather than noise, and that is a statistics question, not a dashboard question.

Telling drift from noise

A raw metric bounces around every day even when nothing is wrong. If you alert on any single bad day you get paged constantly and start ignoring the pager — the classic failure mode called alert fatigue. If you only alert on the multi-week average, you catch the problem only after it has already cost most of the money. What you want is a statistic that accumulates small, persistent shortfalls quickly, while shrugging off noise that doesn't persist.

The standard tool is the cumulative sum, or CUSUM. In words: each day, compare the metric to its target; if it undershoots by more than a small tolerated slack, add the shortfall onto a running total; if it doesn't undershoot, let the running total reset toward zero rather than go negative and cancel out a future problem. Sound the alarm the moment that running total crosses a threshold. Formally,

$$S_t = \max\!\big(0,\; S_{t-1} + (\text{target} - x_t) - k\big),$$

where $x_t$ is today's reading of the metric, the target is what a healthy strategy should produce, $k$ is a small allowance for ordinary noise (so a single so-so day doesn't accumulate), and $S_t$ is the running total. You raise the alarm the first day $S_t$ exceeds a chosen threshold $h$. A bigger $k$ makes the detector more tolerant of noise; a smaller $h$ makes it fire sooner but with more false alarms.

Worked example 1: PnL that still looks fine on average

A strategy targets +3 bps/day. Over five live days it prints +3, +1, −2, 0, −1 bps — noisy, but the five-day average is still +0.2 bps, i.e. barely positive. A team watching only cumulative PnL sees nothing alarming yet. Run the CUSUM with allowance $k = 1$ bp and threshold $h = 8$:

Day	$x_t$	target − $x_t$	$S_t = \max(0, S_{t-1} + (\text{target}-x_t) - k)$
1	3	0	max(0, 0+0−1) = 0
2	1	2	max(0, 0+2−1) = 1
3	−2	5	max(0, 1+5−1) = 5
4	0	3	max(0, 5+3−1) = 7
5	−1	4	max(0, 7+4−1) = 10

$S_5 = 10 > h = 8$: the alarm fires on day 5, while the raw five-day cumulative PnL is still +1 bp. The CUSUM sees the persistent shortfall relative to target that the cumulative total masks, because each mediocre day is compared to what should have happened, not to zero.

Worked example 2: the same machinery on execution quality

CUSUM isn't PnL-specific — it's a generic drift detector, so point it at fill rate too. Target fill rate is 92%; five live days print 90, 88, 85, 91, 80%. With $k = 2$ and $h = 10$:

Day	$x_t$	target − $x_t$	$S_t$
1	90	2	max(0, 0+2−2) = 0
2	88	4	max(0, 0+4−2) = 2
3	85	7	max(0, 2+7−2) = 7
4	91	1	max(0, 7+1−2) = 6
5	80	12	max(0, 6+12−2) = 16

Alarm on day 5 again, this time for a broken order-routing config rather than a broken signal — same detector, different metric, same discipline.

Doing it properly

Wire each panel to its own alert channel and its own owner — a signal-health breach goes to research, a fill-rate breach goes to execution, a data-integrity breach pages whoever owns the feed. Log every alarm and whether it was acted on, because a stack that pages people who then ignore it is worse than no stack: it trains everyone to stop looking. Recompute thresholds periodically against realized false-alarm and miss rates rather than setting them once at launch and forgetting them, and treat the monitoring stack itself as a research artifact — it decays too, just like the strategy it watches.