Squeeze-and-Excitation Blocks

A convolutional layer typically produces dozens or hundreds of output channels, each a different learned feature — one might respond to short-term momentum, another to a volatility spike, another to a spread widening. A plain network treats all channels as equally important all the time, simply adding them together with fixed learned weights. But which channels matter can depend on the current input: on a calm, trending day the volatility-spike channel is irrelevant noise; on a turbulent day it might be the single most informative signal. A squeeze-and-excitation (SE) block lets the network dynamically reweight its channels, input by input, instead of treating channel importance as fixed after training.

The analogy: a mixing board that listens before it adjusts

A sound engineer doesn't set every channel's volume once and leave it — they listen to what's coming through and turn some channels up, others down. An SE block gives a network the same live, input-dependent control: it looks at a rough summary of what each channel currently contains, decides which seem informative for this input, and rescales each channel's output accordingly before passing it on.

The two steps: squeeze, then excite

Squeeze: collapse each channel's entire spatial/time extent to one number — typically its average value (global average pooling). If a layer has $C$ channels, this turns a rich $C$-channel signal into a length-$C$ summary vector $z$, where $z_c$ is roughly "how active was channel $c$, on average, for this input."

Excitation: pass that length-$C$ summary through a small two-layer bottleneck network — shrink to a smaller hidden size, apply a nonlinearity, expand back to $C$, then squash every value into the range $(0,1)$ with a sigmoid:

$$s = \sigma\big(W_2 \, \text{ReLU}(W_1 z)\big)$$

In words: the summary is fed through a small learned network that outputs one number per channel between 0 and 1 — a per-channel "volume knob." Each original channel is then multiplied by its own knob setting $s_c$. Channels judged unhelpful get turned toward 0; channels judged important stay near 1.

That sigmoid curve above is exactly the squashing function used in the excitation step — notice it saturates near 0 and 1 at the extremes, which is why the "volume knobs" it produces cleanly gate a channel almost fully off or leave it almost fully on, rather than hovering ambiguously in the middle.

Worked example 1: squeeze on a tiny 3-channel example

Suppose a layer has 3 channels, and for one input their values across 4 time steps are: channel A $= (0.1, 0.3, 0.2, 0.4)$, channel B $= (0.9, 0.8, 0.85, 0.95)$, channel C $= (0.0, 0.05, -0.05, 0.0)$. Squeeze averages each: $z_A=0.25$, $z_B=0.875$, $z_C=0.0$. Channel B is clearly the most active; channel C is essentially silent.

Worked example 2: excitation gating and its effect

Given $z=(0.25, 0.875, 0.0)$, suppose the excitation network outputs gates $s=(0.4, 0.95, 0.05)$ — it has learned channel B tends to be useful and channel C doesn't, for inputs shaped like this. The rescaled channels become $0.4\times$A, $0.95\times$B, $0.05\times$C — channel C is muted near-zero without changing any convolutional weights that produced it. On a different input where channel C happened to spike (say, a volatility-jump feature on a turbulent day), squeeze would report a high $z_C$ and excitation could output a large $s_C$ instead — the gating is genuinely per-input, not fixed.

What this means in practice

SE blocks are cheap to add (the extra two-layer bottleneck network is tiny relative to the convolutional layers around it) and are typically inserted after a convolutional block, rescaling its output before it's added into a skip connection or passed to the next layer. In a multi-feature financial model, this is a principled way to let the network automatically down-weight a feature channel (say, an illiquid instrument's noisy volume signal) on inputs where it's uninformative, without a human hand-crafting that rule or the feature being discarded entirely for every input.

Practice

1. If the excitation network outputs a gate of exactly $0.5$ for a channel, what does that mean for how much that channel's signal survives into the next layer?
2. Why does the squeeze step use a single average per channel rather than passing the full, un-summarized channel into the excitation network?
3. Explain why an SE block cannot introduce new information that wasn't already present in the original channels — what, precisely, is it allowed to do to them?