Mixup and Input Interpolation

A student who only studies exact past exam questions can ace those questions and still fail a slightly different one, because they memorized answers rather than the underlying pattern. A neural network trained only on its exact training examples has the same failure mode: it can carve out confident, brittle boundaries around each real point without ever being forced to behave sensibly in the space between them.

Mixup manufactures new training examples that live between real ones. For two examples with inputs $x_1, x_2$ and labels $y_1, y_2$, mixup draws a blend weight $\lambda \in [0,1]$ and creates a new example $\tilde{x} = \lambda x_1 + (1-\lambda) x_2$ with a blended label $\tilde{y} = \lambda y_1 + (1-\lambda) y_2$ — the network trains on a weighted average of two inputs and is asked to predict the same weighted average of their labels.

Take a cat image labeled $(1, 0)$ and a dog image labeled $(0, 1)$, blended with $\lambda = 0.7$. The mixed image is 70% cat pixels and 30% dog pixels, and the target label is $(0.7, 0.3)$ — the network is penalized unless its output lands near that split, rather than confidently picking one class. This forces smoother behavior between classes instead of sharp, overconfident boundaries exactly at the training points.