Running A Trading Incident Post-Mortem
How a desk turns a bad trading day, a busted order, a missed hedge, an outage, into a written account of what happened and what changes, instead of a story that gets retold differently each time.
Prerequisites: When The Exchange Or Your Line Goes Down
Something goes wrong on a trading desk, a fat-finger order, a hedge that didn't fire, a feed outage that left a book unmanaged for twenty minutes, and the incident gets fixed, the P&L impact gets booked, and everyone moves on to the next day. Without a deliberate post-mortem, that's often where it ends: the specific fix gets made, but the underlying condition that allowed it stays in place, waiting to produce the same incident again in a slightly different form six months later.
A post-mortem is a short written document, produced soon after the incident while details are fresh, that answers a fixed set of questions regardless of how big or small the incident was: what happened, in a plain timeline; what was the actual impact, in dollars and in anything non-financial like a client or regulatory consequence; what were the immediate causes; and, the part that's easiest to skip, what were the underlying conditions that made this possible, separate from whoever happened to be at the keyboard. Good post-mortems are explicitly not about assigning blame to the individual who was trading when it happened; a culture where post-mortems feel like a disciplinary process is one where people stop reporting near-misses, which is far more dangerous than the incidents themselves.
Worked example
A trader fat-fingers an order size, buying 100,000 shares instead of the intended 10,000, and the error isn't caught for four minutes because the pre-trade size check that should have flagged it had been raised the week before to accommodate a large legitimate order and never reset. The post-mortem timeline shows: order entered at 10:02, size check silently passed at the elevated threshold, position noticed by the trader at 10:06 checking the blotter, unwound by 10:11 at a cost of $14,000 in slippage. The immediate cause is the fat-finger; the underlying condition is a manually-raised risk limit with no automatic expiry or reset reminder. The fix that actually prevents a recurrence isn't "be more careful typing order sizes", it's adding an expiry timestamp to any temporarily-raised limit, so a one-off exception can't silently become the permanent setting.
The output of a good post-mortem is a short list of concrete action items, each with an owner and a deadline, tracked until closed, not just a document that gets filed and forgotten. Incidents that don't produce at least one system, process, or check change are usually a sign the post-mortem stopped at "what happened" and never got to "what allowed it to happen."
A post-mortem separates the immediate cause of an incident (often human, often unavoidable) from the underlying condition that let it cause real damage (usually a process or system gap that can actually be fixed), write it down promptly, blamelessly, and with tracked action items, or the same incident tends to recur.
Write the post-mortem for someone who wasn't in the room, if a new hire six months from now couldn't reconstruct exactly what happened and why from the document alone, it's not detailed enough yet.
Discussion
💡 Discussion rules
- Ask and answer about this concept. Off-topic gets removed.
- No homework dumps. Show what you tried first.
- Corrections are welcome. Cite a source when you claim an error.
Loading discussion…
Related concepts
Practice in interviews
Further reading
- Google SRE Book, ch. 15, Postmortem Culture