Explainability Requirements for Risk Models

A gradient-boosted tail-risk classifier can post excellent backtested performance and still get rejected by a firm's model risk committee, for a reason that has nothing to do with accuracy: nobody can explain why it flags what it flags. In risk management specifically — as opposed to, say, a marketing recommendation engine — the ability to explain a model's output is often not optional polish but a hard requirement, driven both by internal governance and, at regulated institutions, by explicit supervisory guidance that treats an unexplainable model as an unmanaged risk in itself.

An analogy: a doctor who can't explain a diagnosis

A doctor who says "the model says stage 3, trust it" without being able to point to which symptoms or test results drove that conclusion would be malpractice, even if the model is statistically excellent on average — because a patient facing a serious treatment decision needs to know why, to sanity-check it, to catch the cases where the model is wrong for an obvious reason a human would catch immediately. A risk model flagging a portfolio for elevated risk carries the same obligation: a risk manager acting on that flag needs to know which factors drove it, both to sanity-check the flag and to decide what action actually addresses the underlying cause.

What "explainable" concretely means here

For a black-box model like a neural network or gradient-boosted ensemble, explainability usually comes from post-hoc attribution rather than from the model's internal structure being simple. SHAP (Shapley additive explanations) values decompose one specific prediction into the contribution of each input feature, based on a game-theoretic fair-division idea: how much does the prediction change, on average, across every possible subset of features, when this one feature is added in. In plain English, a SHAP value answers "for this specific flagged day, how many points of the risk score came from elevated volatility versus from a correlation spike versus from a liquidity signal" — turning one opaque number into a breakdown a risk manager can act on.

Worked example: reading a SHAP breakdown

An early-warning drawdown score of 78 (on a 0–100 scale, with 70 as the alert threshold) is produced by a gradient-boosted model for a given day. The SHAP breakdown for that prediction shows: baseline score (average across all days) of 25, plus +38 from the 20-day realized volatility feature (which sits at an unusually high percentile that day), plus +12 from the average pairwise correlation feature, plus +3 from a liquidity proxy — summing to 78. A risk manager reading this can immediately tell the flag is being driven overwhelmingly by the volatility spike, not by correlation or liquidity, which changes what response makes sense (a vol-targeting adjustment rather than, say, a liquidity-driven de-risking). Without the breakdown, the manager would only have the number 78 and no way to act on it intelligently.

What this means in practice

Explainability requirements shape model choice from the start — a model that will need clear, per-prediction attribution in production is easier to build around a tree ensemble (where SHAP works efficiently) than around an architecture where attribution is expensive or noisy. At regulated institutions, explainability is often tied directly to formal model risk management frameworks, meaning a model without a working attribution pipeline may not clear governance review regardless of its backtested performance.