One-Hot Encoding and the High-Cardinality Problem

Most models want numbers, but a lot of real features are not numbers — a stock's sector, an order's exchange, a bond's issuer. You cannot feed "Technology" or "NYSE" into a regression as-is, and you cannot just number them 1, 2, 3, because that invents a false ordering: the model would learn that sector 3 is "more" than sector 1 in some sense, which is nonsense for a category with no natural order.

One-hot encoding solves the ordering problem by giving each possible category its own column, filled with 1 where that row belongs to that category and 0 everywhere else. "Technology," "Healthcare" and "Energy" become three separate 0/1 columns instead of one column coded 1, 2, 3. No column implies an ordering, and the model can weight each category completely independently.

Worked example

A dataset of five stocks has a sector column with values Technology, Healthcare, Technology, Energy, Healthcare. One-hot encoding replaces that single column with three:

stock	sector_Technology	sector_Healthcare	sector_Energy
A	1	0	0
B	0	1	0
C	1	0	0
D	0	0	1
E	0	1	0

A linear model can now give each sector its own coefficient. If Technology stocks tend to have higher beta, the model learns a positive weight on sector_Technology — a relationship it had no way to express through a single 1/2/3-coded column.

Where it breaks: cardinality

This works cleanly when a category has a handful of values. It stops working when the category has thousands — a ticker column across the whole US equity universe, a counterparty field in a trading system, a CUSIP. One-hot encoding a 5,000-ticker universe creates 5,000 mostly-empty columns. Three problems follow directly from that:

1. Dimensionality explosion. The feature matrix balloons, most models slow down or need far more data to fit stably, and distance-based methods (k-NN, anything using cosine similarity) get dominated by the sheer count of sparse dimensions rather than the signal in the dense ones.
2. Sparsity per category. If most tickers appear only a handful of times, the model sees almost no examples per one-hot column, so any weight it learns for a rare ticker is mostly noise — a direct route to Overfitting.
3. New categories at prediction time. A ticker that IPOs after training gets no column at all; the encoding has nowhere to put it.

The alternatives

Target encoding replaces each category with a summary statistic of the outcome variable for that category — a ticker's historical average return, for instance — collapsing thousands of columns into one. The danger is leakage: computing a ticker's average outcome using rows that include the row you are predicting leaks the label into the feature. The fix is to compute the encoding out-of-fold, the same discipline used in stacking, and to shrink rare categories' statistics toward the overall average (regularizing toward the population mean) so a ticker with three observations does not get a wildly noisy encoding.

Embeddings learn a small dense vector per category jointly with the model, the same idea neural networks use for words. Tree-based models often handle raw categorical codes directly and can split on groups of categories without needing one-hot columns at all, which is one reason gradient-boosted trees dominate high-cardinality tabular finance problems — see Gradient Boosting in Finance.

Hashing and grouping as cheaper middle grounds

Two lighter-weight fixes sit between full one-hot and learned encodings. Feature hashing maps every category through a fixed hash function into a small number of buckets — say, 200 — instead of one column per category; it is fast, requires no fitting step, and handles brand-new categories automatically (a new ticker just hashes into an existing bucket). The cost is occasional hash collisions, where two unrelated categories land in the same bucket and the model can no longer fully separate their effects. Manual grouping — bucketing individual tickers into their GICS sector, or grouping rare counterparties into an "other" category below some frequency threshold — is the crudest fix and often the most robust one, because it encodes a real prior rather than relying on the data alone to find structure in categories it has barely seen.

Worked example: the new-category problem concretely

A model trained on five years of history has a ticker one-hot encoding with 4,200 columns. A company IPOs six months after training ends and needs a prediction. One-hot encoding has no column for it — every one of the 4,200 columns is correctly zero, indistinguishable from "not any of these tickers," and the model has learned nothing about how to weight a ticker it never encoded. A target-encoded or hashed pipeline degrades more gracefully: target encoding falls back to the population mean for an unseen ticker, and hashing simply routes it into whichever bucket its hash lands in, inheriting that bucket's learned behaviour instead of defaulting to nothing at all.