ResNet and Identity Mappings

Take a network that works well at 20 layers and add 15 more, purely by stacking on identity-like layers that in principle could just pass their input straight through unchanged. Common sense says performance can only stay the same or improve — worst case, the new layers learn to do nothing. In practice, before ResNet, it got worse: deeper plain networks had higher training error than their shallow counterparts, not from overfitting but because they became measurably harder to optimize. Gradients flowing back through many stacked layers of matrix multiplies and nonlinearities shrink or explode, and a plain deep stack simply couldn't find the "do nothing extra" solution even though it was sitting right there in its own hypothesis space.

The analogy: a moving walkway with shops along the side

Picture an airport corridor. A moving walkway carries you straight to the far end at a fixed speed, no effort required. Alongside it are optional shops — detours that add something to your day (a coffee, a magazine) but that you only take if they're worth the stop. If a shop is useless, you just skip it and the walkway still gets you there. Compare that to a corridor with no walkway, only a chain of shops you're forced to enter one after another; skipping a bad one means somehow threading through it anyway, and a long chain of forced stops makes the whole trip fragile. A residual network is the corridor with the walkway: each block's default behavior is "pass through unchanged," and the learned layers only need to figure out the extra adjustment on top, not reconstruct the whole signal from scratch.

The maths

A plain network layer computes a direct mapping from input to output:

$$y = F(x)$$

In words: whatever function $F$ the layer's weights represent, that is the entire output — the layer must learn the complete transformation the network needs at that point, including simply preserving $x$ if that's the right answer.

A residual block instead computes:

$$y = F(x) + x$$

In words: run $x$ through a small stack of learned layers to get $F(x)$ — the residual, or "correction" — then add the original input back in via a skip connection (also called a shortcut). If the ideal transformation at this depth really is "do nothing," the network only has to learn $F(x) = 0$, which is trivially easy — push the weights toward zero — rather than learning $F(x) = x$ exactly, which requires the layer to reconstruct an identity mapping out of nonlinear pieces, a much harder target to hit by gradient descent.

The payoff shows up in backpropagation. Differentiating $y = F(x) + x$ with respect to $x$:

$$\frac{\partial y}{\partial x} = \frac{\partial F}{\partial x} + 1$$

In words: the gradient flowing backward through a residual block is whatever the learned layers contribute, plus a guaranteed "+1" from the skip connection. Even if $\partial F/\partial x$ shrinks toward zero — the classic vanishing-gradient failure — the gradient signal still has that clean +1 path straight through, so it doesn't die as it's chained back through dozens or hundreds of blocks.

Worked example 1: gradient flow through 3 blocks, plain vs residual

Suppose each weight layer's local gradient, $\partial F/\partial x$, happens to be a modest $0.4$ at every block (a plausible value once activations pass through repeated nonlinearities). Chain three plain layers by multiplying:

$$0.4 \times 0.4 \times 0.4 = 0.064$$

Only 6.4% of the original gradient signal survives three layers back. Now chain three residual blocks, where each local derivative is $(0.4 + 1) = 1.4$:

$$1.4 \times 1.4 \times 1.4 = 2.744$$

The residual gradient doesn't vanish — it doesn't even shrink. In a real 50- or 100-layer network the effect compounds: a plain stack's gradient decays toward zero roughly geometrically, while the residual version's floor is anchored by the chain of +1's, so early layers keep receiving a usable training signal no matter how deep the stack goes.

Worked example 2: a numeric forward pass

Let $x = 2.0$ and suppose the block's two weight layers, before training has done anything useful, compute $F(x) = 0.1x^2 - 0.05x$ (a stand-in for whatever the learned layers currently express). Then:

$$F(2.0) = 0.1(4) - 0.05(2) = 0.4 - 0.1 = 0.3$$ $$y = F(x) + x = 0.3 + 2.0 = 2.3$$

Compare a plain layer that had to learn the whole mapping from scratch with the same starting weights: it would just output $F(x) = 0.3$, badly wrong if the target really was close to 2. The residual block's output, 2.3, stays anchored near the input even while $F$ is still random and untrained — the network starts from a sensible baseline and only has to nudge away from it, rather than starting from an arbitrary point and having to find the entire right answer through training.

Try a very steep, narrow loss surface and watch how a poorly conditioned landscape makes it easy for steps to overshoot or crawl. That's the optimization difficulty ResNet's identity shortcuts sidestep — they reshape the loss surface near "do nothing" into something flatter and easier for gradient descent to navigate at extreme depth.

What this means in practice

Skip connections are now the default in almost every deep architecture that goes beyond a handful of layers, including transformers (every attention and feed-forward sub-block has one) and U-Nets. For a quant training a deep network on tick-level order-book features or a long price history, "just add skip connections" is close to a free way to make a deeper network actually trainable — it costs one extra addition per block and no extra parameters. It's also why architecture ablations that report "removing skip connections drops accuracy by double digits" aren't overselling; the shortcuts aren't a minor refinement, they're what makes the optimization tractable at all past a certain depth.