Highway Networks

Stacking many neural network layers on top of each other tends to make training harder rather than easier: gradients shrink as they're pushed back through many layers, so early layers barely update and adding depth can actually hurt performance. Highway networks address this by giving each layer a choice, learned from data, between transforming its input (passing it through the usual weights and activation) and simply passing the input through unchanged. A "gate" — itself a small learned function of the input — decides, per unit, how much of each option to use, producing an output that's a weighted mix: mostly the transformed version where the gate says transform, mostly the raw input where the gate says skip.

This gating mechanism (borrowed from LSTM-style recurrent networks) means gradients can flow straight through the "skip" path during backpropagation without being squeezed by many layers of transformation, which is what let researchers train networks tens of layers deep at a time when doing so otherwise stalled. Highway networks were a direct conceptual predecessor to residual networks (ResNets), which simplified the idea by using a fixed unweighted skip connection instead of a learned gate, and residual connections went on to become the more widely used default in modern deep architectures.

Think of it like a highway with occasional exits: most traffic (information) can just stay on the highway (skip the layer's transformation) and reach its destination unchanged, while only the traffic that needs a specific detour (real transformation) takes an exit. The gate is the sign at each exit deciding how much traffic to divert. Concretely, if $x$ is a layer's input, $H(x)$ its usual transformed output, and $T(x)$ a gate function outputting a value between 0 and 1, the layer's output is $y = T(x)\cdot H(x) + (1-T(x))\cdot x$ — a weighted blend of "transform" and "skip," where the weight itself is learned rather than fixed. When $T(x)$ is near 0, the layer behaves almost like a pass-through, and gradients flowing backward during training pass through nearly untouched, avoiding the vanishing-gradient problem that made very deep plain networks hard to train.