Running A Trading Incident Post-Mortem
How a desk turns a bad trading day — a busted order, a missed hedge, an outage — into a written account of what happened and what changes, instead of a story that gets retold differently each time.
Prerequisites: When The Exchange Or Your Line Goes Down
Something goes wrong on a trading desk — a fat-finger order, a hedge that didn't fire, a feed outage that left a book unmanaged for twenty minutes — and the incident gets fixed, the P&L impact gets booked, and everyone moves on to the next day. Without a deliberate post-mortem, that's often where it ends: the specific fix gets made, but the underlying condition that allowed it stays in place, waiting to produce the same incident again in a slightly different form six months later.
A post-mortem is a short written document, produced soon after the incident while details are fresh, that answers a fixed set of questions regardless of how big or small the incident was: what happened, in a plain timeline; what was the actual impact, in dollars and in anything non-financial like a client or regulatory consequence; what were the immediate causes; and — the part that's easiest to skip — what were the underlying conditions that made this possible, separate from whoever happened to be at the keyboard. Good post-mortems are explicitly not about assigning blame to the individual who was trading when it happened; a culture where post-mortems feel like a disciplinary process is one where people stop reporting near-misses, which is far more dangerous than the incidents themselves.
Worked example
A trader fat-fingers an order size, buying 100,000 shares instead of the intended 10,000, and the error isn't caught for four minutes because the pre-trade size check that should have flagged it had been raised the week before to accommodate a large legitimate order and never reset. The post-mortem timeline shows: order entered at 10:02, size check silently passed at the elevated threshold, position noticed by the trader at 10:06 checking the blotter, unwound by 10:11 at a cost of $14,000 in slippage. The immediate cause is the fat-finger; the underlying condition is a manually-raised risk limit with no automatic expiry or reset reminder. The fix that actually prevents a recurrence isn't "be more careful typing order sizes" — it's adding an expiry timestamp to any temporarily-raised limit, so a one-off exception can't silently become the permanent setting.
The output of a good post-mortem is a short list of concrete action items, each with an owner and a deadline, tracked until closed — not just a document that gets filed and forgotten. Incidents that don't produce at least one system, process, or check change are usually a sign the post-mortem stopped at "what happened" and never got to "what allowed it to happen."
A post-mortem separates the immediate cause of an incident (often human, often unavoidable) from the underlying condition that let it cause real damage (usually a process or system gap that can actually be fixed) — write it down promptly, blamelessly, and with tracked action items, or the same incident tends to recur.
Write the post-mortem for someone who wasn't in the room — if a new hire six months from now couldn't reconstruct exactly what happened and why from the document alone, it's not detailed enough yet.
Related concepts
Practice in interviews
Further reading
- Google SRE Book, ch. 15, Postmortem Culture