Contractive Autoencoders

A standard autoencoder learns to compress data into a smaller representation and then reconstruct the original, but nothing in that setup stops the learned representation from being wildly sensitive to tiny, meaningless input changes — a slightly noisy version of the same input could map to a very different code. A contractive autoencoder fixes this by adding a penalty term during training that directly punishes how much the encoded representation changes when the input is nudged slightly, on top of the usual reconstruction error.

Practically, this penalty is computed from the encoder's Jacobian — how sensitive each output of the encoding is to each input feature — and keeping it small forces nearby inputs to map to nearby representations. The result is a representation that emphasizes the directions in the data that actually matter for reconstruction while being deliberately flat, or "contractive," in directions that just reflect noise.

Worked example. Two nearly identical price series differ only by a small amount of quote noise. A standard autoencoder might encode them to noticeably different feature vectors; a contractive autoencoder's penalty specifically discourages this, pulling their encoded representations closer together and making downstream models trained on those features less sensitive to noise.