Word Embeddings and Word2Vec

Count-based text features like Bag of Words and TF-IDF treat every word as its own unrelated column. To that representation, profit and earnings are as different as profit and lawsuit — each is just a separate slot in a very wide, very sparse table. So a model trained on filings that say "strong profit" has learned nothing about filings that say "strong earnings". You end up needing enormous amounts of labelled text to relearn, one synonym at a time, things a human reader already knows.

Embeddings fix this by giving each word a position instead of a slot. Picture a map where every word in the vocabulary is a pin. Words that get used in the same kinds of sentences are pinned close together; words used in unrelated contexts are pinned far apart. Once words have coordinates, "close in meaning" becomes "close in distance", and a model that has seen one word gets partial credit for the others nearby.

The rule that builds the map is old and simple: you shall know a word by the company it keeps. If two words repeatedly appear surrounded by the same neighbours, they probably mean similar things. Nobody has to label anything — the raw text supplies its own supervision.

How Word2Vec learns the map

Word2Vec turns the "company it keeps" idea into a prediction task. In the skip-gram version, you slide a window over the text — say five words wide — and for each centre word you ask the model to predict its neighbours. Given the centre word dovish, predict that Fed, statement, rates are nearby.

Each word gets two vectors, one for when it is the centre and one for when it is a neighbour, and the model scores a (centre, neighbour) pair by their dot product:

$$\text{score}(c, n) = v_c \cdot v_n$$

In plain English: two words score highly together when their coordinate lists point in the same direction. Those scores are turned into probabilities and the vectors are nudged by gradient descent so that pairs which really co-occur score high, and randomly sampled pairs which do not score low. That last trick — training against a handful of random "negative" words instead of the whole vocabulary — is called negative sampling, and it is what makes the whole thing fast enough to run on billions of words.

Nothing in that procedure knows what dovish means. The meaning is an emergent property of where the word lands.

Worked example: measuring similarity

Distance alone is a poor measure because common words get long vectors. What matters is direction, so similarity is measured by the cosine of the angle between two vectors:

$$\cos(u, v) = \frac{u \cdot v}{\lVert u \rVert \, \lVert v \rVert}$$

In plain English: divide out the lengths and ask only whether the two arrows point the same way. The result runs from $-1$ to $1$.

Take three toy three-dimensional embeddings: $\text{profit} = (2,\, 1,\, 0)$, $\text{earnings} = (3,\, 2,\, 0.5)$, and $\text{lawsuit} = (-1,\, 2,\, 3)$.

For profit and earnings, the dot product is $2(3) + 1(2) + 0(0.5) = 8$. The lengths are $\lVert \text{profit} \rVert = \sqrt{5} = 2.236$ and $\lVert \text{earnings} \rVert = \sqrt{13.25} = 3.640$, so

$$\cos = \frac{8}{2.236 \times 3.640} = \frac{8}{8.139} = 0.983 .$$

Nearly parallel — the model treats them as close to interchangeable. For profit and lawsuit, the dot product is $2(-1) + 1(2) + 0(3) = 0$, giving a cosine of exactly $0$: perpendicular, meaning no relationship at all. That single number is what powers duplicate-headline detection, peer-firm mapping, and the retrieval step in Vector Databases and Embedding Search.

Worked example: relationships as arrows

The famous property of Word2Vec is that consistent relationships become consistent displacements. Using two-dimensional stand-ins: $\text{share} = (1,1)$, $\text{shares} = (1,2)$, $\text{bond} = (3,1)$.

The singular-to-plural move is $\text{shares} - \text{share} = (0,\, 1)$. Apply the same move elsewhere:

$$(3,\,1) + (0,\,1) = (3,\,2) .$$

Now look up the nearest actual word to $(3,2)$ in the vocabulary. If the embedding is any good, it is bonds. The model was never told about plurals; it discovered that plural contexts differ from singular ones in a consistent way, and encoded that difference as a direction. The same trick recovers company-to-ticker and currency-to-country pairs in financial corpora.

Embeddings are also the point where noise enters quietly. Two headlines about different companies can sit close together simply because they share boilerplate, so cosine similarity on raw text needs the same care as any other alpha feature.