Pooling Layers and Downsampling

A convolutional network builds up feature maps — grids of numbers describing what patterns were detected where — and these grids stay the same size as the input unless something shrinks them. Pooling layers do exactly that: they slide a small window (commonly $2 \times 2$) across a feature map and replace each window with a single summary value, cutting the map's height and width in half (or more) each time a pooling layer is applied.

The most common variant, max pooling, simply keeps the largest value in each window, on the logic that the strongest activation in a small neighborhood is the most informative one — if a feature was strongly detected anywhere nearby, that's what matters, not its exact pixel position. Average pooling instead takes the mean of the window, smoothing rather than picking a peak. Max pooling is more common in practice because it tends to preserve sharp, distinctive features better.

Shrinking the feature map serves two purposes at once: it drastically cuts the number of computations needed in later layers (a network working on progressively smaller grids needs far fewer multiplications), and it makes the network somewhat tolerant to small shifts in where a feature appears in the image, since a feature detected a pixel or two away from where it was in training will usually still fall in the same pooling window and produce the same output. Modern architectures increasingly replace pooling with strided convolutions that learn their own downsampling instead of using a fixed rule, but pooling remains common, especially in smaller or older network designs.