Explainability Requirements in Regulated Finance
Financial regulators often require that an automated decision affecting a customer come with a specific, verifiable reason — a constraint that shapes which models are usable in production.
In many jurisdictions, a lender who denies a consumer credit application must give the applicant specific, accurate reasons for the denial — in the US this is required under the Equal Credit Opportunity Act's adverse action notice rules. That legal requirement runs headlong into modern credit models: a gradient-boosted tree ensemble or neural network can be highly predictive but doesn't naturally produce a short list of "the top reasons this application was declined" the way a simple scorecard does. Regulators have generally not accepted "the model said so" as an adequate reason, so firms using complex models must attach an explanation method — commonly SHAP values or a similar attribution technique — that translates the model's output into a small set of named factors a compliance officer can sign off on and a customer can be told.
This creates a real engineering constraint, not just a nice-to-have: the explanation method has to be stable enough that the same application always gets the same top reasons, fast enough to run at application volume, and accurate enough that the stated reasons genuinely reflect what drove the score, since a mismatched explanation is itself a compliance risk.
Regulations like the US adverse action notice requirement force lenders to attach an explanation method to any credit model complex enough to lack one natively, turning explainability from an academic nicety into a hard production requirement.
Further reading
- US Federal Reserve, Regulation B / Equal Credit Opportunity Act adverse action notice requirements