The Hashing Trick

A ticker universe with 8,000 symbols, a broker-ID field with new counterparties added weekly, a free-text order-tag column — all of these are categorical variables where one-hot encoding either blows up your feature matrix into tens of thousands of columns or breaks the moment a new category shows up in production that the training set never saw. You need a fixed-width numeric representation that doesn't require ever building or storing a lookup table of every category you might encounter.

The idea: bucket by hash, not by lookup

Instead of assigning each category its own column by looking it up in a table built from training data, feed the category's name (as a string) through a hash function and take the result modulo some fixed number of buckets, $m$. "AAPL" might hash to bucket 41, "GOOG" to bucket 8, a counterparty called "NEWBROKER2026" that never appeared during training hashes to bucket 41 too — deterministically, with no dictionary lookup at all. The feature vector has exactly $m$ columns no matter how many distinct categories exist in the data, now or in the future.

Formally, for a hash function $h$ mapping strings to integers, the bucket index is

$$b = h(\text{category}) \bmod m .$$

In plain English: pick a fixed number of buckets $m$, run every category name through the same hash function, and the remainder after dividing by $m$ tells you which bucket to light up. Two different categories that happen to land in the same bucket are said to collide — the model can no longer fully distinguish them, and this collision noise is the price paid for a fixed, unbounded-vocabulary-proof feature width.

Worked example: hashing four tickers into 8 buckets

Suppose $m = 8$ and a (illustrative) hash function gives: $h(\text{AAPL}) = 41$, $h(\text{MSFT}) = 17$, $h(\text{GOOG}) = 8$, $h(\text{TSLA}) = 33$. Taking each mod 8: AAPL → bucket 1 ($41 \bmod 8 = 1$), MSFT → bucket 1 ($17 \bmod 8 = 1$), GOOG → bucket 0, TSLA → bucket 1. AAPL, MSFT and TSLA all collide into bucket 1 — the model sees a single "bucket-1 indicator" turn on for any of the three tickers and cannot tell them apart from that feature alone. With only 8 buckets and 4 tickers, collisions are likely; doubling $m$ to 16 or 32 sharply reduces the chance any two specific tickers collide, at the cost of a wider feature vector.

What this means in practice

The hashing trick is the standard fix for high-cardinality or open-ended categoricals — order-tag free text, exchange venue codes, counterparty IDs — where a lookup-table encoding would either be too wide or fail on unseen values at inference time. Choosing $m$ is a direct bias-collision trade-off: too small and unrelated categories get merged into meaningless shared signal; too large and you're back to a near one-hot-sized matrix with little benefit. A common check is estimating the birthday-paradox-style collision rate for your actual vocabulary size and picking $m$ several times larger than the number of distinct categories you expect.