The Trading Incident Response Runbook

When a live trading system starts misbehaving — a data feed goes stale, a strategy sends an unexpectedly large order, a reconciliation break turns out to be a real error rather than a timing quirk — the worst time to figure out what to do is in the middle of it. An incident response runbook is a document written in advance, while everyone is calm, that lays out exactly what steps to take for each category of problem: who to notify, what to check first, how to safely halt trading if needed, and who has the authority to make each kind of decision. The point is to convert a stressful, high-stakes improvisation into a rehearsed checklist.

A good runbook is organized by incident type rather than as one generic document, because "the data feed died" and "an order for ten times the intended size got sent" call for very different first steps. Each entry typically specifies: how to recognize this particular incident (what alert or symptom triggers it), the immediate containment action (often: stop new orders, but not necessarily unwind existing positions blindly), who must be paged and in what order, and what information needs to be logged for the post-incident review. Crucially, a runbook should specify who has authority to halt trading without needing sign-off first — during an actual incident is not the time to be waiting on an email reply from someone senior.

A concrete example: an automated strategy sends an order for 50,000 shares instead of the intended 5,000, due to a decimal-parsing bug in a configuration file. The runbook's entry for "unexpected order size" instructs the on-call engineer to immediately kill the strategy's order-sending process (a single documented command), then check whether the erroneous order filled or can still be canceled, then page the strategy owner, and only after those steps investigate the root cause. Because the sequence was rehearsed, the entire containment took under two minutes; without a runbook, the same incident might have taken twenty minutes of people arguing about the right order of operations while the position kept growing.