Intrinsically Interpretable vs Post-Hoc Explained Models

A risk committee asks why a model flagged a particular trade as high-risk. There are two fundamentally different ways to answer that question, and they are not interchangeable. One is: "the model is a formula simple enough that you can read the answer directly off its coefficients." The other is: "the model is too complex to read directly, so a separate technique estimated, after the fact, roughly which inputs mattered most." Both get called "explainable AI," but only the first one is telling you what the model actually computed.

Intrinsically interpretable models

A model is intrinsically interpretable when a human can trace its exact reasoning from inputs to output without any auxiliary tool. A linear regression's coefficients tell you, precisely, how much the prediction moves per unit change in each feature — that number is the model's logic, not an approximation of it. A shallow decision tree's path of if/then splits is likewise the literal computation, readable top to bottom. Logistic regression, generalized additive models, and small rule lists all belong to this category: transparency is a structural property of the model, not something bolted on.

Post-hoc explained models

A model is post-hoc explained when the model itself — a gradient-boosted ensemble of hundreds of trees, a deep neural network — is too complex for a human to trace directly, so a separate explanation technique is run against it to produce something interpretable-looking: a SHAP value per feature, a saliency map, a counterfactual example. Crucially, the explanation is a model of the model — a simplified approximation of how the real model behaves near a given point — not the model's actual internal computation. This is a meaningfully weaker guarantee, and it is easy to lose sight of that distinction once the explanation comes out looking as clean as a coefficient table.

post-hoc explained 500-tree ensemble SHAP / saliency approximation, not the model

Worked example

A credit-scoring model rejects an applicant. Version A is logistic regression: the loan officer can say precisely, "your debt-to-income ratio contributed $-1.2$ to the log-odds, your payment history contributed $+0.4$," and those numbers are literally what the model computed — recomputing them by hand from the coefficients and the applicant's data reproduces the decision exactly. Version B is a 300-tree gradient-boosted model; a SHAP explanation says debt-to-income "contributed roughly $-1.1$" to this applicant's score, estimated by comparing the ensemble's output with and without that feature across many resampled backgrounds. That number is a good-faith estimate of influence near this particular applicant's data point, not a component the model literally summed. Change the SHAP background sample or the estimation method and the number can shift measurably; change the logistic regression's coefficient and you have changed a different model.

The trade-off is real, but often smaller than assumed

The usual argument for reaching straight for a black-box model is that flexibility buys accuracy. That is sometimes true, but on many structured, tabular finance problems — credit scoring, churn, fraud flags — a carefully feature-engineered linear or additive model comes within a small margin of a full gradient-boosted ensemble's accuracy, while keeping a real, exact explanation instead of an approximated one. The choice is worth making deliberately rather than defaulting to complexity: see Gradient Boosting in Finance for where the extra flexibility does earn its keep, and Regularization as a Prior for how much of an intrinsic model's apparent simplicity comes from deliberately constraining it.

Where the middle ground actually sits

The distinction is not a strict binary in practice. Generalized additive models (GAMs) sit between the two extremes: each feature gets its own flexible, potentially non-linear curve rather than a single coefficient, so the model can capture curved relationships a plain linear model cannot, while each curve can still be plotted and read directly — the explanation remains exact, just no longer a single number per feature. Shallow gradient-boosted ensembles (a handful of trees, each only a few levels deep) are a similar middle case: too complex to trace by hand in full, but small enough that a post-hoc explanation of them tends to be far more faithful and stable than the same technique applied to a five-hundred-tree ensemble, because there is simply less complexity for the approximation to misrepresent.

Faithfulness as the deciding property

When a post-hoc explanation is unavoidable — the underlying model is complex enough that no intrinsically interpretable alternative comes close on accuracy — the property to demand from the explanation technique is faithfulness: does the explanation's account of what mattered actually match how the model's output changes when you intervene on those features directly, rather than merely correlate with them. This is precisely the test that distinguishes a genuinely useful post-hoc method like Shapley-based attribution, which is built around a principled decomposition of the prediction, from weaker heuristics that merely look plausible. It is the same standard Attention Is Not Explanation shows attention weights routinely fail.