Unseen Categories at Inference Time

A model trained on last year's universe of 500 tickers will, sooner or later, be asked to score ticker number 501 — a new listing, a newly-covered name, a category that simply did not exist when the encoding was built. Most categorical encoding schemes have no natural answer to "what number do I use for something I've never seen," and what happens by default is often silent and wrong, not a helpful error message.

The analogy: a form with a dropdown that's missing an option

Imagine a paper form with a dropdown for "country of residence," printed the year before three new countries joined the list. Someone from one of those new countries fills out the form and simply cannot select their real answer — they either leave it blank, pick the closest-sounding existing option, or the form processor crashes. None of those outcomes are "correct," and which one happens depends entirely on how the form (or the software reading it) was designed to handle a value it never anticipated. A categorical encoder facing an unseen category at inference time is in exactly that position, and the fallback behavior has to be decided in advance, not discovered in production.

What actually happens, encoder by encoder

One-hot encoding represents each known category as its own column; an unseen category has no matching column at all. Naively, this either crashes (a strict pipeline) or silently produces an all-zero row — which many models will interpret as "matches none of the known categories," a specific and sometimes misleading signal rather than an honest "unknown."

Target encoding replaces a category with its average target value; an unseen category has no such average to look up. The standard fallback is to use the global prior mean $p$ — exactly the same "zero pseudo-observations" endpoint that Bayesian smoothing already produces for a category with $n=0$:

$$\text{enc}(\text{unseen}) = \frac{0 \cdot \bar{y} + m \cdot p}{0 + m} = p$$

In plain English: with literally no observations of the new category, the smoothing formula from Bayesian Smoothing for Rare Categories collapses cleanly to "just use the population-wide average" — which is a principled fallback, not a special case bolted on afterward, as long as smoothing was used in the first place.

Frequency/hashing-based encodings (The Hashing Trick) assign every possible string — seen or not — to some bucket via a hash function, so an unseen category simply lands wherever its hash points, colliding with whatever known category already occupies that bucket, without ever raising an error.

Worked example 1: an all-zero row from naive one-hot encoding

A model trained with one-hot columns for sectors {Tech, Energy, Healthcare, Financials} sees a new row for a newly-covered sector, "Utilities." Naive one-hot encoding produces the row $[0,0,0,0]$ — identical, numerically, to a hypothetical row that legitimately belonged to none of the four known sectors, which cannot actually happen for a well-formed input. A linear model reading that all-zero row treats it as a specific, meaningful pattern (the intercept term alone, with no sector adjustment), silently and without any signal that this is actually a missing-category situation rather than a real data point.

Worked example 2: target encoding degrading gracefully

A ticker that IPO'd yesterday has zero historical target observations. Using the smoothed formula with $p=0.50$ (global up-day rate) and $m=20$: $\text{enc} = (0 \times \bar{y} + 20 \times 0.5)/20 = 0.50$ exactly — the model treats the brand-new ticker as "average," which is a reasonable, honest default, versus a naive unsmoothed target encoder that would either error out or, worse, silently divide by zero and return NaN, which many pipelines then propagate into a broken prediction without any warning.

What this means in practice

The safest default across encoders is to reserve an explicit "unknown" category during training — a designated bucket the encoder is taught to use for rare or held-out categories even when every training category was technically seen, so the unseen-at-inference path is exercised and validated before it ever matters in production. This matters constantly in quant pipelines: new listings, delistings, corporate actions creating new tickers, and reclassified sectors all generate categories a model was never trained on.