Learned Embeddings for Categorical Features

A stock's sector is a category — "Technology," "Utilities," "Energy" — not a number. Feed it to a model as one-hot encoding (a column of zeros with a single 1 marking which sector) and every sector becomes equally different from every other. The model is told nothing about the fact that Technology and Communication Services often move together, while Technology and Utilities usually don't. One-hot vectors are all the same distance apart, by construction, whether the categories are close cousins or total opposites.

An embedding fixes that by learning, from the data itself, where each category should sit. Instead of "Technology" being a single 1 among five hundred zeros, it becomes a short list of numbers, say eight of them, and the model learns those eight numbers the same way it learns any other weight — by gradient descent on the training loss. If sectors that behave alike end up needing similar predictions, gradient descent will naturally pull their embeddings close together, because that's what lowers the loss. Think of it like a seating chart at a large conference that nobody drew up in advance: people who keep ending up in the same conversations gradually migrate to sit near each other, and by the end of the event, physical closeness on the chart tells you who has similar interests, purely as a byproduct of many small social decisions.

The mechanics, one symbol at a time

Say a categorical feature has $K$ possible values (e.g., $K=11$ sectors). An embedding table is a matrix $E$ with $K$ rows, one per category, and $d$ columns, where $d$ (the embedding dimension) is a size you choose, typically much smaller than $K$. For a category $c$, its embedding is simply that row:

$$e_c = E_{c,:} \in \mathbb{R}^d$$

In words: look up row $c$ of a table, and the $d$ numbers you find there are that category's learned representation. There's no formula computing $e_c$ from anything else — the table's entries are free parameters, initialized randomly and then updated by backpropagation exactly like any weight matrix, based on how much moving $e_c$ would have reduced the loss on examples where category $c$ appeared.

Worked example 1: a two-dimensional embedding lookup

Suppose $d = 2$ and the embedding table for four sectors, after training, has settled at:

sector	$e_1$	$e_2$
Technology	1.8	0.9
Communication Services	1.6	1.1
Utilities	-1.5	0.2
Energy	-1.3	-0.8

Technology's row is looked up as $e_{\text{Tech}} = (1.8, 0.9)$, and that pair of numbers, not a one-hot vector, is what's concatenated with the stock's other features (say, momentum and volatility) before being fed to the rest of the network.

Compute the (Euclidean) distance between Technology and Communication Services: $\sqrt{(1.8-1.6)^2 + (0.9-1.1)^2} = \sqrt{0.04+0.04} = \sqrt{0.08} \approx 0.28$, close together. Between Technology and Energy: $\sqrt{(1.8-(-1.3))^2 + (0.9-(-0.8))^2} = \sqrt{9.61+2.89} = \sqrt{12.5} \approx 3.54$, far apart. Nobody told the model sectors should cluster this way; it emerged because Technology and Communication Services examples needed similar predictions during training, while Energy examples needed very different ones.

Worked example 2: why one-hot loses information a small model can't recover

Ten sectors, one-hot encoded, feeding a linear layer with weight $w_i$ per sector. The layer's contribution for sector $i$ is just $w_i$ — ten completely independent numbers with zero sharing between them. If the smallest sector has only 40 training examples, that sector's $w_i$ is estimated from 40 points alone.

With a $d=2$ embedding instead, that same small sector's prediction is a function of two numbers sitting inside a continuous space shared by every sector. Gradient descent, when nudging Technology's embedding based on thousands of Technology examples, also slightly reshapes the local geometry near sectors that started close to it — so a data-starved sector inherits some signal from its neighbors, rather than learning everything from its own 40 examples in isolation. This is the practical payoff: embeddings let rare categories borrow statistical strength from similar, common ones.

The explorer above shows a different setting (a matrix stretching the unit circle) but the same underlying idea applies: a $K \times d$ embedding table is a linear map from a high-dimensional one-hot space down to a low-dimensional space, and training bends that map so that geometric closeness in the small space means learned similarity, not an accident of alphabetical category order.

What this means in practice

Embedding dimension $d$ is a hyperparameter — too small re-creates the "everything looks similar" problem for genuinely different categories, too large wastes parameters and risks overfitting a category with few examples. A common rule of thumb scales $d$ with $K$ (something like $d \approx \min(50, \lceil K^{0.5}\rceil)$), but it's worth tuning per problem. Embeddings need enough examples per category to be learned well; a category seen five times will end up with a nearly-arbitrary embedding, so rare categories are sometimes grouped into an explicit "other" bucket first. The same core idea underlies word embeddings — a ticker, a word, a merchant category are all just categories waiting for a learned coordinate.