U-Net: Encoder-Decoder With Skip Connections

Some tasks need output at the same resolution as the input, but built from global understanding: label every pixel of a medical scan as tumor or not, denoise every point of a volatility surface while respecting the overall shape of the smile, forecast every minute of an order-book snapshot while conditioning on the whole session's regime. A plain encoder that compresses the input down to a small bottleneck and a decoder that expands it back up struggles here — every time you downsample, you throw away exactly the fine-grained detail (the sharp edge, the single spiky quote, the exact strike) that the final output needs to reproduce. The bottleneck captures the big picture but is a lossy funnel for detail, and no amount of decoder cleverness can invent detail that was thrown away three layers earlier.

The analogy: tracing paper over a rough sketch

An architect blocks out a building's overall shape on a small napkin sketch — just the rough massing, nothing precise. To turn that into a construction drawing, she doesn't work from the napkin alone; she lays tracing paper over her original, detailed site survey and uses the napkin's shape as a guide for where the big structures go, while copying exact boundary lines straight from the survey underneath. The final drawing combines coarse guidance (from the compressed sketch) with fine detail (copied directly from the original, at the same points). A U-Net does exactly this: its decoder gets both the compressed "what's the overall shape" signal from the bottleneck and a direct copy of the fine detail from the matching-resolution layer on the way down, spliced in via a skip connection.

The maths

A U-Net has two halves. The encoder (contracting path) repeatedly applies convolutions and downsampling (usually max-pooling), halving spatial resolution at each stage while doubling the number of channels:

$$h_\ell = \text{Pool}(\text{Conv}(h_{\ell-1}))$$

In words: at stage $\ell$, run a convolution over the previous stage's output $h_{\ell-1}$, then shrink it — this builds a stack of feature maps at successively coarser resolutions, ending in a small, information-dense bottleneck.

The decoder (expanding path) reverses this, upsampling back toward the original resolution:

$$u_\ell = \text{Conv}\big(\text{concat}[\text{Upsample}(u_{\ell+1}),\ h_\ell]\big)$$

In words: take the decoder's previous (coarser) output $u_{\ell+1}$, upsample it back to stage $\ell$'s resolution, then concatenate it — stack it channel-wise, side by side, not add it — with $h_\ell$, the encoder's output at that exact same resolution, saved earlier via the skip connection. Run a convolution over the combined stack. The key symbol is $h_\ell$: it is a value computed early in the forward pass and carried, unchanged, all the way across the network to be used late in the forward pass. Nothing about it passes through the bottleneck.

Worked example 1: why concatenation beats "decoder alone"

Suppose a 1-D toy input has 8 points, and one of them is a sharp spike: $x = (1, 1, 1, 9, 1, 1, 1, 1)$. Downsample by average-pooling in pairs:

$$\text{pool} = (1.0,\ 5.0,\ 1.0,\ 1.0)$$

Already, the spike's exact location and height are smeared into the value 5.0 shared with its neighbor. Pool again to 2 points, and the spike is barely a bump in an average. A decoder trying to reconstruct the original 8-point series from that heavily pooled bottleneck alone has to guess where the spike belongs and how tall it truly was — it can only recover a blurry approximation.

Now suppose the encoder saved $h_1 = (1.0, 5.0, 1.0, 1.0)$ — the first pooling stage's output — as a skip connection. When the decoder upsamples the bottleneck back to 4 points and concatenates $h_1$, it has direct access to the actual pooled value 5.0 at the right position, not a doubly-blurred guess. Each additional skip level recovers one more level of the detail that a bottleneck-only path destroyed. That's the entire mechanism, just repeated at every resolution.

Worked example 2: counting channels through a skip

Say encoder stage 2 outputs $h_2$ with 128 channels at a $32\times32$ resolution. The decoder's matching stage upsamples the previous decoder output (256 channels at $16\times16$) to $32\times32$, keeping 256 channels. Concatenating gives:

$$256 + 128 = 384 \text{ channels at } 32\times32$$

A convolution at that decoder stage must therefore be built to accept 384 input channels, not just the 256 the upsampling alone produced — this is the single most common shape bug when implementing a U-Net from scratch: forgetting that concatenation adds channel counts, so every decoder convolution's input-channel dimension must be sized for "upsampled channels + skip channels," not just one or the other.

A convolution's learned filters are just structured linear maps; watching how a transform stretches, rotates, and can lose information along a collapsed axis is a useful mental model for what repeated pooling does to spatial detail — and why the skip connection's job is to hand back exactly the axis the bottleneck flattened away.

What this means in practice

U-Net-style architectures show up anywhere the required output is dense and full-resolution while still needing global context: denoising or completing a volatility surface, segmenting anomalous regions in a heatmap of order-flow imbalance, or reconstructing a corrupted price panel. The architecture is a strict superset of a plain autoencoder — same bottleneck idea, plus the skip highway — so anywhere you'd reach for an autoencoder but the output needs to preserve fine local structure that the input had, a U-Net is very likely the better default.