Amortized Inference and Encoder Networks

Classical variational inference re-derives the approximate posterior distribution over latent variables separately for every single data point, running an optimization loop each time — accurate, but expensive to repeat across a large dataset, since each new observation starts the optimization from scratch. Amortized inference changes the trade: instead of solving that optimization freshly per data point, train one neural network (the "encoder" or "inference network") whose job is to take any input and directly output the parameters of its approximate posterior in a single forward pass, learned once across the whole dataset.

The "amortized" name reflects exactly this economics: the network's training cost is a one-time upfront investment, spread ("amortized") across every future inference call, each of which then costs only a forward pass rather than a fresh optimization. The trade-off is a loss of per-point accuracy — the encoder gives an approximation that's only as good as the shared function it learned, so it can systematically underfit certain data points that don't look like the training distribution, sometimes called the "amortization gap."

The variational autoencoder is the canonical use of this idea: its encoder network maps any input image or observation directly to the mean and variance of an approximate posterior over the latent code, replacing what would otherwise be a separate gradient-based optimization for every single training example.