Interpretability vs Explainability

These two words get used almost interchangeably in casual conversation, but they describe fundamentally different situations. Interpretability means the model itself is simple enough that a person can trace through exactly how it produced a given output — a linear regression's coefficients literally are the reasoning; a shallow decision tree's path literally is the decision. There is no gap between "what the model computed" and "what you understand the model did," because they're the same thing.

Explainability applies to models where that's not true — a gradient-boosted ensemble of a thousand trees, a deep neural network — and it means applying a separate technique after the fact to approximate why the model behaved a certain way. Methods like SHAP or LIME don't reveal the model's actual internal computation; they build a simplified, local approximation of it and report that approximation's reasoning instead. The explanation is a model of the model, not the model itself, which means it can be wrong, incomplete, or misleading about what really happened inside the black box, even when it looks clean and confident on the page.

Worked example

A credit-scoring quant compares two approaches to a loan-approval model. The first is a scorecard: a weighted sum of five factors (income, debt ratio, payment history, and so on), each with a published coefficient. When a loan is denied, the reason is literally "your debt ratio contributed −12 points and your payment history contributed −8, together crossing the threshold" — that sentence describes the actual arithmetic the model performed, because the model is that arithmetic. The second approach is a gradient-boosted tree ensemble that scores slightly better on held-out accuracy. To explain a denial, the team runs SHAP on the specific application and gets a similar-looking breakdown: "debt ratio contributed −11, payment history contributed −9." But this number is an approximation constructed by perturbing the input and observing how the ensemble's output moves around that one point — it did not read the ensemble's actual decision path, because a thousand-tree ensemble making a decision by averaging doesn't have a single decision path to read. Both explanations may agree in this case, but only the first is guaranteed to be exactly what happened; the second is a best estimate that could diverge for a different applicant, especially near a decision boundary.

explainable (black box) tree ensemble computation SHAP approximation

What this means in practice

The choice matters most in regulated or high-stakes settings — credit, hiring, anything a client or regulator might dispute — where "we ran an explanation technique that suggests these factors mattered" is a weaker guarantee than "here is the exact rule that produced this decision." Some researchers argue for defaulting to interpretable models whenever the accuracy gap over a black box is small, precisely because post-hoc explanations aren't the same guarantee. In lower-stakes settings, like an internal signal-monitoring dashboard, the accuracy of a black box plus a good-enough post-hoc explanation is often the right trade-off.