Kill Switches and Circuit Breakers

In 2012, a bad software deployment left an old, unused piece of code active on one of Knight Capital's trading servers. It started sending unintended orders into the market, and because there was no automated way to stop it fast enough, it lost the firm about $440 million in 45 minutes before anyone could pull the plug by hand. The lesson quant desks took from that: no strategy, however well-tested, should be trusted to run without a mechanism that can stop it faster than a human can diagnose what's wrong. That mechanism is a kill switch — a way to halt trading immediately — and a circuit breaker is its automated form, tripping on its own when a predefined limit is breached, without waiting for a human to notice.

The idea: independent, simple, and triggered by outcomes not intentions

Two design choices make a kill switch actually work under stress. First, it must be independent of the strategy's own code — if the bug is in the strategy, you cannot trust the strategy to correctly decide it's buggy and stop itself. A separate risk-gateway process, sitting between the strategy and the exchange, checks every order against hard limits before it's allowed through.

Second, it triggers on observed outcomes, not on inferred intentions: not "does this look like the strategy's normal behavior" (subjective, arguable, slow) but "has P&L dropped more than $X today," "has order rate exceeded Y per second," "has position size exceeded Z" — bright-line numbers anyone can check in microseconds.

class RiskGateway:
    def __init__(self, max_daily_loss, max_order_rate, max_position):
        self.max_daily_loss = max_daily_loss
        self.max_order_rate = max_order_rate
        self.max_position = max_position
        self.halted = False

    def check(self, pnl_today, orders_last_second, position):
        if self.halted:
            return False
        if (pnl_today < -self.max_daily_loss
                or orders_last_second > self.max_order_rate
                or abs(position) > self.max_position):
            self.halted = True          # trip the breaker; stays tripped
            return False
        return True                     # order allowed through

Worked example: a runaway order loop tripped

A strategy has a bug in a retry loop and starts sending an order every 2 milliseconds instead of the intended once every 2 seconds — a 1,000x speedup nobody noticed in testing because the bug only appears under a rare timing condition. The risk gateway's max_order_rate is set to 50 orders/second. Within the first second, orders_last_second climbs past 50, check() returns False, self.halted flips to True, and every subsequent order from the strategy — buggy or not — is rejected at the gateway before it ever reaches the exchange. The strategy process itself may still be looping, unaware anything is wrong; it doesn't need to be aware, because the gateway sits between it and the market and simply stops passing orders through.

Where it shows up

Interviewers ask "how would you design a kill switch" as a systems-design question with no single right answer, but strong answers name concrete trip conditions (daily loss limit, order-rate limit, position limit, and a stale-data check — no market data update in N seconds should also halt), name where the check lives (a gateway process, not the strategy), and address the reset path (does it require a human to manually re-arm, which is almost always the right default, since an automatic reset can just re-trigger the same bug).

In production, kill switches exist at multiple layers: exchange-level circuit breakers halt an entire market on extreme volatility, firm-level risk gateways halt one firm's order flow, and strategy-level switches halt one strategy — see Kill Switch Scope: What Actually Gets Cancelled for how these layers interact and why the scope of a halt (one strategy vs. the whole book vs. the whole firm) is itself a design decision. New strategies typically prove themselves in shadow mode before their kill switch is trusted with real capital (see Paper Trading and Shadow Mode).