Subgroup Performance and Fairness Slices

A credit-approval model's overall approval accuracy looks excellent, but a closer look shows its false-rejection rate for applicants from one region is triple the rate for applicants from another, despite similar underlying default risk in the historical data. This isn't just an ethical or regulatory problem, though it is that too — it's also a sign the model has learned to lean on a feature or a data artifact that correlates with the subgroup rather than with true creditworthiness, which is a genuine modeling flaw, not only a fairness one. Subgroup performance evaluation is the practice of computing performance metrics separately within demographically or operationally meaningful groups, rather than trusting one blended number for the whole population.

What to check, concretely

For a chosen grouping variable (region, age band, account tenure, customer segment — whatever is meaningful and, where relevant, legally protected), compute the same metrics used for the overall model — true positive rate, false positive rate, precision, calibration — separately within each group, and compare them side by side:

$$\text{Gap} = \max_g \; m(g) - \min_g \; m(g)$$

where $m(g)$ is a chosen metric (say, false-rejection rate) computed within group $g$. In plain English: rather than one overall number, look at the spread across groups — a small gap suggests the model treats groups consistently; a large gap flags a subgroup being systematically over- or under-served relative to the others, which needs investigating even if the average across all groups looks fine.

Worked example: uncovering a hidden proxy

A model approving small-business loans shows an overall false-rejection rate of 8%. Split by applicant zip code income tercile: low-income zip codes see a 19% false-rejection rate, middle-income 7%, high-income 4%. The gap between the lowest and highest group is 15 percentage points — far larger than the blended 8% figure suggests. Digging into the feature importances reveals the model leans heavily on "years at current address," which correlates with zip code income tercile through housing turnover patterns rather than with actual repayment behavior. The subgroup breakdown didn't just reveal a fairness gap — it pointed directly at the specific feature causing it, which a single aggregate accuracy number gave no hint of at all.

Overlaying separate subgroup distributions of a metric like predicted default probability (rather than one pooled distribution) is a similar diagnostic — a visible shift between two groups' curves is the same signal as a numeric gap, made easier to see at a glance.

What this means in practice

Subgroup evaluation should be standard practice before deploying any model that makes decisions about people — credit, hiring, insurance pricing, fraud flags — both because regulators increasingly require it (disparate impact analysis) and because a large subgroup gap is often a tell that the model is exploiting a spurious correlate of a protected characteristic rather than a genuine risk driver. Where a meaningful gap is found, the fix is rarely "just remove the protected attribute" — it's usually still encoded indirectly through proxies, so the fix has to address the underlying feature or process causing the disparity.