Residual Connections

Stack enough ordinary layers and, counterintuitively, training gets worse, not just slower. This isn't overfitting — a 56-layer plain network can have higher training error than a 20-layer one on the exact same data. The reason is what happens to the gradient during backpropagation: it has to travel back through every layer, getting multiplied by that layer's weights each time. Multiply enough small numbers together and the signal that reaches the early layers is vanishingly faint; those layers barely learn anything, and the whole network is stuck.

Imagine relaying a message through fifty people, each one only allowed to whisper their own edited version onward — by person fifty the original message is unrecognizable static. A residual connection is the fix: hand each person a copy of the original note alongside whatever they whisper, so the message never depends entirely on fifty consecutive accurate retellings.

The mechanics, one symbol at a time

Let $x$ be a layer's input and $F(x)$ be whatever transformation the layer's weights compute (its ordinary forward pass — some matrix multiplies and nonlinearities). A plain layer outputs $F(x)$. A residual layer instead outputs:

$$y = x + F(x)$$

In words: don't ask the layer to produce the whole answer — just ask it to produce the change needed on top of what came in, and add that change back to the original input. If the ideal transformation for this layer happens to be "do nothing," the layer only has to learn $F(x) = 0$, which is far easier for gradient descent to find than learning the identity mapping exactly using ordinary weighted sums and nonlinearities.

The gradient benefit follows directly. Differentiating $y = x + F(x)$ with respect to $x$ gives $\frac{\partial y}{\partial x} = 1 + \frac{\partial F}{\partial x}$. That leading $1$ means a direct, unobstructed path for the gradient straight back through the skip connection, no matter how small or poorly conditioned $\frac{\partial F}{\partial x}$ is. The signal from the loss can always reach earlier layers through that "+1" highway, even if the layer's own learned transformation is contributing almost nothing useful yet.

Worked example 1: a forward pass with and without the skip

A single residual block, input $x = 2.0$. Its internal transformation is $F(x) = 0.1x^2 - 0.3$.

$F(2.0) = 0.1(4) - 0.3 = 0.4 - 0.3 = 0.1$.

Plain layer output: $y_{\text{plain}} = F(x) = 0.1$ — the input's own scale is gone entirely; downstream layers see only the small correction.

Residual layer output: $y_{\text{res}} = x + F(x) = 2.0 + 0.1 = 2.1$ — the input's information survives, lightly adjusted. If $F$ is badly initialized and produces something close to zero or noise, the plain layer passes that noise onward as its entire output, while the residual layer still passes a nearly-unchanged, sensible $x$.

Worked example 2: gradient flow through a chain

Chain three plain layers where each layer's local gradient happens to be small, $\frac{\partial F_i}{\partial x} = 0.2$ at the current weights (this happens easily early in training or with saturating activations). The gradient reaching the input, by the chain rule, is roughly $0.2 \times 0.2 \times 0.2 = 0.008$ — the signal has shrunk by more than 99%, and the earliest layer barely updates.

Now make each layer residual, so each local derivative is $1 + 0.2 = 1.2$ instead. Chained: $1.2 \times 1.2 \times 1.2 = 1.728$. The gradient not only survives three layers, it doesn't shrink at all. This is the concrete mechanism behind "residual connections fix vanishing gradients": the additive $+1$ in every layer's derivative stops the chain rule from being a chain of pure multiplication.

Use the explorer above with a base below 1 to see how fast repeated multiplication by a number less than one collapses toward zero — that curve, applied to gradients instead of a variable, is exactly what happens to a plain deep network and exactly what the residual's $+1$ interrupts.

What this means in practice

Residual connections are what made networks with 50, 100, even 1000+ layers trainable, and they are now a default architectural choice, not a special trick — they sit inside every transformer block (around both the attention step and the feed-forward step) and every modern convolutional network. One practical requirement: $x$ and $F(x)$ must have the same shape to be added, so architectures using residual connections are built with that constraint in mind, sometimes inserting a small linear projection on the skip path when a layer changes dimensionality. Financial deep learning models are rarely hundreds of layers deep, so you will meet residual connections less as "the reason training works at all" and more as a component inherited from transformer-based architectures used for sequence data.