Bayesian Smoothing for Rare Categories

Target encoding replaces a category — say, an exchange ticker — with the average outcome for that category, like average next-day return for stocks on that exchange. That works fine for categories with thousands of observations, but a category seen only twice, both times happening to be up days, gets encoded as "100% chance of an up day" — a wildly overconfident number produced from essentially no evidence. Bayesian smoothing fixes exactly this: it pulls small-sample averages back toward a sensible baseline, and lets large-sample averages speak almost entirely for themselves.

The analogy: a new restaurant with two five-star reviews

A restaurant with two reviews, both five stars, has a raw average of 5.0 — better, on paper, than a restaurant with 4,000 reviews averaging 4.6. Nobody actually believes the two-review restaurant is better; two reviews are not enough evidence to overturn the assumption that most restaurants land somewhere in the ordinary range. A sensible rating system blends the raw average with a prior belief about typical restaurant quality, weighted by how much evidence there is — two reviews barely move the needle from the prior, four thousand reviews move it almost entirely to the observed average. That blend, formalized, is Bayesian smoothing.

The formula

For a category with $n$ observations and observed target average $\bar{y}_{\text{cat}}$, and an overall prior average $p$ (typically the global target mean), the smoothed estimate is

$$\text{enc} = \frac{n \cdot \bar{y}_{\text{cat}} + m \cdot p}{n + m}$$

where $m$ is a smoothing strength — think of it as "how many pseudo-observations of the prior to blend in." In plain English: treat the prior as if it were $m$ additional, imaginary observations all equal to $p$, mix them in with the real observations, and average everything together. When $n \gg m$, the real data dominates and $\text{enc} \approx \bar{y}_{\text{cat}}$. When $n \ll m$, the prior dominates and $\text{enc} \approx p$.

Worked example 1: two ratios shown side by side

Global average target $p = 0.50$ (say, 50% of days are up days across the whole universe), smoothing strength $m=20$. Category A: $n=2$, $\bar{y}_A = 1.0$ (both up days). Smoothed: $(2 \times 1.0 + 20 \times 0.5)/(2+20) = (2+10)/22 = 0.545$ — barely above the prior, correctly reflecting that two observations are almost no evidence. Category B: $n=5{,}000$, $\bar{y}_B = 0.58$. Smoothed: $(5000 \times 0.58 + 20 \times 0.5)/(5020) = (2900+10)/5020 = 0.5797$ — essentially unchanged from the raw average, because 5,000 observations overwhelm 20 pseudo-observations of prior.

Worked example 2: choosing $m$ changes how cautious the encoding is

Keep category A ($n=2$, $\bar{y}_A=1.0$, $p=0.5$) but vary $m$. At $m=1$: $(2+0.5)/3 = 0.833$ — barely smoothed, still overconfident. At $m=20$: $0.545$, as above. At $m=200$: $(2+100)/202 = 0.505$ — almost entirely the prior, since 2 real observations are swamped by 200 pseudo-observations. Choosing $m$ is choosing how many observations you require before you start trusting a category's own average over the population-wide baseline — a direct, interpretable knob, not an arbitrary tuning constant.

The explorer above shows a Bayesian posterior blending a prior with observed data exactly the same way — drag the data count up and watch the posterior pull away from the prior and toward the observed rate, which is the same mechanism smoothing rare categories relies on.

What this means in practice

Any target-encoded categorical feature in a quant pipeline — sector, exchange, counterparty, instrument type — needs this smoothing, because financial categories are almost always unevenly represented: a handful of large-cap tickers dominate observation counts while many small or newly-listed names have only a few dozen data points. Skipping smoothing routinely produces target-encoded features that look powerful in training and fall apart out of sample, because the model has effectively memorized noise for rare categories.