DenseNet and Feature Reuse

In a standard deep convolutional network, each layer only sees the output of the layer directly before it, so any useful feature computed early on has to be re-derived or passed along, layer by layer, if a much later layer needs it — and that signal tends to weaken or get overwritten as it travels through many layers. DenseNet's fix is architectural: instead of connecting each layer only to the one before it, every layer receives the concatenated outputs of all preceding layers within its block as input.

Concretely, layer $l$'s input is not just layer $l-1$'s output but the concatenation of the outputs of layers $0, 1, \ldots, l-1$ — every feature ever computed in the block stays directly accessible to every later layer, rather than being filtered through a single narrow pipeline. This has two practical effects: feature reuse, since a low-level feature (an edge, a texture) computed early can be reused directly by a much later layer without having to be re-learned, and improved gradient flow during training, since the gradient during backpropagation has a short, direct path back to every earlier layer instead of having to travel back through every intervening layer's transformation.

Because concatenating every layer's full output would make each layer's input grow very wide (and expensive) very fast, DenseNet layers typically add only a small, fixed number of new feature channels — the "growth rate" — at each step, so the total input width grows linearly with depth rather than exploding, and periodic "transition layers" compress and downsample the accumulated features between blocks.

The tradeoff against similarly-deep architectures using simple skip connections (like ResNet) is that DenseNet tends to need fewer total parameters to reach comparable accuracy, since so much computation is genuinely reused rather than duplicated across layers, at the cost of higher memory use from keeping all those earlier feature maps around during training.