Contrastive Self-Supervised Learning

Supervised learning needs labels, and labels are expensive — someone had to decide, by hand, what the "right answer" is for every training example. Most raw data in the world has no labels at all: years of unlabeled tick data, unlabeled news text, unlabeled satellite images. Contrastive learning is a way to train a genuinely useful network on that unlabeled data by inventing its own task, one that needs no human annotation whatsoever.

The trick: create two different, distorted "views" of the same underlying example — say, two overlapping crops of one chart image, or one bar of returns and a slightly noised copy of it — and train the network to recognize that they came from the same source, while also recognizing that a view from a completely different example is not the same source. It's like teaching someone to recognize a person from photos taken at different angles, lighting, and distance: they never need a label saying "this is person X," they just need to learn "these two photos, despite looking different, are the same person" versus "these two photos are different people." Do that across millions of photo pairs and person-recognition ability emerges as a byproduct, with the labels of who each person is never entering the picture.

The mechanics, one symbol at a time

For an example $x$, create two augmented views $x_i$ and $x_j$ (say, two noised or cropped versions), and pass both through the same encoder to get representations $z_i = f(x_i)$, $z_j = f(x_j)$. Call $(x_i, x_j)$ a positive pair (same underlying source) and every other example in the batch a source of negative pairs. A common contrastive loss, InfoNCE, for one positive pair is:

$$\mathcal{L}_{i,j} = -\log \frac{\exp(\text{sim}(z_i, z_j)/\tau)}{\sum_{k} \exp(\text{sim}(z_i, z_k)/\tau)}$$

In words: $\text{sim}(\cdot,\cdot)$ is a similarity score between two representations (typically cosine similarity — how aligned two vectors are, close to 1 if they point the same way). The numerator rewards the positive pair for being similar; the denominator sums similarity to every other example in the batch (the negatives), including the positive itself. Minimizing this loss means pushing the positive pair's similarity up relative to all the negatives — the representation of $x_i$ should be closer to its true partner $x_j$ than to any unrelated example $k$. $\tau$ (the temperature) controls how sharply the model is punished for near-misses: a small $\tau$ makes the loss much harsher on negatives that are almost as similar as the positive, forcing tighter separation.

Worked example 1: computing the loss for a small batch

A batch has one positive pair with cosine similarity $\text{sim}(z_i, z_j) = 0.8$, and two negatives with similarities $\text{sim}(z_i, z_k) = 0.2$ and $\text{sim}(z_i, z_m) = 0.1$. Temperature $\tau = 0.5$.

Scaled similarities: positive $0.8/0.5 = 1.6$, negatives $0.2/0.5=0.4$ and $0.1/0.5=0.2$.

Exponentiate: $e^{1.6}\approx 4.95$ (positive, appears in both numerator and denominator), $e^{0.4}\approx 1.49$, $e^{0.2}\approx 1.22$.

Denominator: $4.95 + 1.49 + 1.22 = 7.66$.

$$\mathcal{L} = -\log\frac{4.95}{7.66} = -\log(0.646) \approx 0.437$$

Now suppose training has pulled the positive pair much closer, $\text{sim}(z_i,z_j) = 0.98$, everything else unchanged: scaled $0.98/0.5=1.96$, $e^{1.96}\approx 7.10$. Denominator $7.10+1.49+1.22=9.81$. Loss: $-\log(7.10/9.81) = -\log(0.724) \approx 0.323$, lower — the loss falls as the positive pair becomes relatively more similar than the negatives, exactly the direction training pushes it.

Worked example 2: the role of temperature

Same scenario as above ($\text{sim}$ values $0.8, 0.2, 0.1$) but with a much smaller temperature, $\tau = 0.1$: scaled similarities become $8, 2, 1$. Exponentials: $e^8 \approx 2981$, $e^2\approx 7.39$, $e^1\approx 2.72$. Denominator $\approx 2991$.

$$\mathcal{L} = -\log\frac{2981}{2991} = -\log(0.9967) \approx 0.0033$$

A far smaller loss than $\tau=0.5$ gave for the identical similarities — a low temperature makes an already-decent separation look almost perfect to the loss function, so gradients shrink and the model coasts. Push $\tau$ too low the other direction (very tiny) and gradients for genuinely close negatives explode instead. This is why $\tau$ is a sensitive hyperparameter in practice: it rescales how harshly "almost as similar as the positive" is punished, and getting it wrong either starves training of signal or destabilizes it.

The exponentials inside InfoNCE mean small changes in similarity, after dividing by a small $\tau$, translate into large changes in the loss — the explorer above's exponential curve is exactly this amplification, and it's why temperature has an outsized effect on training dynamics.

What this means in practice

In quant work, contrastive pretraining is used to build reusable representations of price series, limit-order-book snapshots, or news text from vast unlabeled history, before fine-tuning a small labeled head on a specific supervised task — the same transfer-learning pattern used in vision and language, applied to market data where labels (like "this was profitable" or "this preceded a crash") are far scarcer than raw data itself. The quality of the learned representation depends heavily on the choice of augmentations: for a chart image, cropping and slight noise are natural "same underlying thing" views, but for financial time series it is genuinely unclear what a label-preserving augmentation even is — shuffle too aggressively and you destroy the temporal structure that made the series meaningful in the first place, which is an open and actively debated design problem in applying contrastive learning to finance.