Autoencoders

Give a model a task where the "answer key" is the input itself — reproduce this exact vector of numbers you were just shown — and it sounds trivial; the identity function does it perfectly. But force the model to pass that input through a narrow bottleneck first, a layer with far fewer numbers than the original, and the trivial solution is no longer available. It can't just copy; it has to compress, and the only way to compress well is to figure out what's actually important in the data and discard the rest.

That's an autoencoder: a packing exercise. Imagine trying to describe a full outfit — shirt, pants, shoes, jacket, accessories — using only three words, then handing those three words to someone else who has to reconstruct the outfit from scratch. You're forced to identify the three facts that matter most (maybe "formal," "navy," "winter") and let everything else be inferred or lost. A good compression scheme emerges only because the bottleneck makes sloppy compression impossible.

The mechanics, one symbol at a time

An autoencoder has two halves. The encoder $f$ maps the input $x$ down to a low-dimensional latent code $z$; the decoder $g$ maps $z$ back up to a reconstruction $\hat{x}$:

$$z = f(x), \qquad \hat{x} = g(z), \qquad \mathcal{L} = \| x - \hat{x} \|^2$$

In words: encode, then decode, then measure how far the reconstruction $\hat{x}$ landed from the original $x$ using squared error (add up the squared difference in every dimension). Training just means adjusting the weights of $f$ and $g$ together to make that loss as small as possible — there are no labels anywhere in this process, only the input compared against its own reconstruction, which is why autoencoders are called unsupervised. The dimension of $z$ (the bottleneck's width) is a hyperparameter you choose: make it too wide and the network can cheat toward the identity function without learning anything useful; make it appropriately narrow and it's forced to find real structure.

Worked example 1: compressing two correlated features to one

Two input features that happen to be almost perfectly correlated in the training data: momentum $x_1$ and 3-month trailing return $x_2$, which in practice move together (correlation near 0.95) because they measure overlapping things. Suppose a trained linear encoder learns $z = 0.7x_1 + 0.7x_2$ (roughly capturing "average trend strength") and the decoder learns $\hat{x}_1 = 0.7z$, $\hat{x}_2 = 0.7z$.

For an input $(x_1, x_2) = (2.0, 1.8)$: $z = 0.7(2.0) + 0.7(1.8) = 1.4 + 1.26 = 2.66$.

Reconstruction: $\hat{x}_1 = 0.7(2.66) = 1.86$, $\hat{x}_2 = 0.7(2.66) = 1.86$.

Loss: $(2.0-1.86)^2 + (1.8-1.86)^2 = 0.0196 + 0.0036 = 0.023$ — small, because one number, $z=2.66$, was almost enough to reconstruct both original numbers. The bottleneck forced the network to notice the two features carry mostly the same information and collapse them into a shared summary — the same thing principal component analysis does, but learned by gradient descent rather than by an eigenvalue calculation, and capable of nonlinear compression if the encoder and decoder have hidden layers with nonlinear activations.

Worked example 2: reconstruction error flags an unusual input

Train the same autoencoder only on "normal" days, where momentum and trailing return move together. Now feed it an unusual day where the two features have decoupled: $(x_1, x_2) = (2.0, -1.5)$ — momentum strongly positive, trailing return strongly negative, a combination the network never saw enough of during training.

Using the same encoder: $z = 0.7(2.0) + 0.7(-1.5) = 1.4 - 1.05 = 0.35$.

Reconstruction: $\hat{x}_1 = 0.7(0.35) = 0.245$, $\hat{x}_2 = 0.7(0.35) = 0.245$.

Loss: $(2.0-0.245)^2 + (-1.5-0.245)^2 = 3.08 + 3.05 = 6.13$ — over 250 times larger than the normal-day loss of 0.023. The single learned direction $z$ simply cannot represent two features moving in opposite directions, because it never had to during training. A large reconstruction error is a direct, quantitative signal that this input doesn't fit the pattern the network learned — which is exactly the mechanism behind using autoencoders for anomaly and regime-break detection.

Squared reconstruction error grows the way the curve above does: small deviations from what the network has learned to reconstruct well cost very little, but the penalty accelerates fast as an input becomes more unusual, which is what makes reconstruction error such a sensitive anomaly signal.

What this means in practice

Autoencoders are used for dimensionality reduction ahead of another model, denoising (train to reconstruct a clean version from a noisy input, and the bottleneck forces it to learn the underlying signal rather than the noise), and anomaly detection, exactly as in worked example 2 — a spike in reconstruction error on live data flags a regime that doesn't resemble training history. Unlike PCA, an autoencoder's encoder and decoder can be nonlinear, so it can capture curved, more complex structure that a straight-line projection misses, at the cost of needing more data and careful tuning of the bottleneck width to avoid either under-compressing (bottleneck too wide, nothing useful is learned) or over-compressing (bottleneck too narrow, even normal reconstructions are poor).