Backprop Through a Two-Layer Network by Hand

Reading about backpropagation and actually pushing real numbers through a real network by hand are two different levels of understanding, and only the second one leaves you able to spot a bug in someone else's implementation. This page does exactly that: one small network, two inputs, two hidden neurons, one output, every single number written down, forward and backward, with a check at the end that proves the gradients are correct.

The analogy: a two-stage supply chain

A raw material's price moves. To know how much that move affects the price of a finished product, you cannot look at the raw material and the finished product alone — the effect passes through an intermediate processing stage first. If a $1 rise in the raw material adds$2 to the cost of an intermediate part, and a $1 rise in that intermediate part's cost adds$3 to the finished product, then the raw material's total effect on the finished product is $2 \times 3 = 6$ dollars, found by multiplying the effect at each stage. A two-layer network has exactly this structure: an input affects a hidden neuron, and that hidden neuron affects the output, and the input's total effect on the output is the product of the two local effects — chained twice instead of once.

The network

Two inputs $x_1 = 1.0$, $x_2 = 0.5$. Two hidden neurons, each a weighted sum of the inputs passed through a sigmoid, $\sigma(z) = 1/(1+e^{-z})$:

$$z_1 = w_{11}x_1 + w_{12}x_2 + b_1, \quad a_1 = \sigma(z_1) \qquad\qquad z_2 = w_{21}x_1 + w_{22}x_2 + b_2, \quad a_2 = \sigma(z_2)$$

with weights $w_{11}=0.3,\ w_{12}=-0.2,\ b_1=0.1$ and $w_{21}=0.4,\ w_{22}=0.1,\ b_2=-0.05$. A single linear output neuron combines the two hidden activations: $\hat y = v_1 a_1 + v_2 a_2 + c$, with $v_1 = 0.5$, $v_2 = -0.3$, $c = 0.2$. Target $y = 1.0$, loss $L = \frac{1}{2}(\hat y - y)^2$.

Worked example 1: the full forward and backward pass

Forward pass, computed and kept for later:

$$z_1 = 0.3(1.0) - 0.2(0.5) + 0.1 = 0.300, \qquad a_1 = \sigma(0.300) = 0.5744$$ $$z_2 = 0.4(1.0) + 0.1(0.5) - 0.05 = 0.400, \qquad a_2 = \sigma(0.400) = 0.5987$$ $$\hat y = 0.5(0.5744) - 0.3(0.5987) + 0.2 = 0.2872 - 0.1796 + 0.2 = 0.3076$$ $$L = \tfrac{1}{2}(0.3076 - 1)^2 = \tfrac{1}{2}(0.6924)^2 = 0.2397$$

Backward pass. Start at the output, whose local slope is $\partial L/\partial \hat y = \hat y - y = 0.3076 - 1 = -0.6924$. Because the output layer is linear, that is also $\delta_3 = \partial L/\partial z_3 = -0.6924$.

Output weights — each output weight's gradient is $\delta_3$ times the hidden activation it multiplied:

$$\frac{\partial L}{\partial v_1} = \delta_3\, a_1 = -0.6924(0.5744) = -0.3977, \qquad \frac{\partial L}{\partial v_2} = \delta_3\, a_2 = -0.6924(0.5987) = -0.4145, \qquad \frac{\partial L}{\partial c} = \delta_3 = -0.6924$$

Into the hidden layer — each hidden unit's incoming gradient is $\delta_3$ times the output weight it fed into, times its own sigmoid's local slope $a(1-a)$:

$$\delta_1 = \delta_3\, v_1\, a_1(1-a_1) = (-0.6924)(0.5)(0.5744)(0.4256) = -0.0846$$ $$\delta_2 = \delta_3\, v_2\, a_2(1-a_2) = (-0.6924)(-0.3)(0.5987)(0.4013) = 0.0499$$

Notice $\delta_2$ came out positive even though $\delta_3$ is negative, because $v_2$ is negative — the second hidden neuron helps the output by shrinking, and backprop finds that sign flip automatically, exactly as it must.

Hidden weights — each hidden weight's gradient is its unit's $\delta$ times the input it multiplied:

$$\frac{\partial L}{\partial w_{11}} = \delta_1 x_1 = -0.0846, \quad \frac{\partial L}{\partial w_{12}} = \delta_1 x_2 = -0.0423, \quad \frac{\partial L}{\partial b_1} = \delta_1 = -0.0846$$ $$\frac{\partial L}{\partial w_{21}} = \delta_2 x_1 = 0.0499, \quad \frac{\partial L}{\partial w_{22}} = \delta_2 x_2 = 0.0250, \quad \frac{\partial L}{\partial b_2} = \delta_2 = 0.0499$$

Nine numbers, six weights and three biases, all obtained from one forward pass and one backward sweep — never re-running the network from scratch for each one.

Worked example 2: verifying two gradients by finite difference

Nudge $v_1$ up by $0.001$ to $0.501$, holding everything else fixed, and redo only the parts of the forward pass that change: $\hat y' = 0.501(0.5744) - 0.3(0.5987) + 0.2 = 0.3082$, giving $L' = \frac{1}{2}(1 - 0.3082)^2 = 0.2393$.

$$\frac{L' - L}{0.001} = \frac{0.2393 - 0.2397}{0.001} = -0.392$$

That is within rounding of the backprop value $\partial L/\partial v_1 = -0.3977$ computed above — a genuine, independent check, not a restatement of the same calculation. Now nudge $w_{11}$ up by $0.001$ to $0.301$ instead: $z_1' = 0.301$, $a_1' = \sigma(0.301) = 0.5747$, $\hat y' = 0.5(0.5747) - 0.3(0.5987) + 0.2 = 0.3077$, $L' = \frac{1}{2}(1-0.3077)^2 = 0.2396$.

$$\frac{L' - L}{0.001} = \frac{0.2396 - 0.2397}{0.001} = -0.084$$

Matching backprop's $\partial L/\partial w_{11} = -0.0846$. Both checks land within finite-difference rounding error of the analytic values, confirming the backward sweep is correct — and this is exactly the gradient-checking routine every hand-rolled implementation should run once before trusting it.

Every one of the nine gradients above feeds into exactly the kind of step the explorer above shows: a small move opposite the gradient's direction, one number at a time. Drag the step size and watch how the same mechanics that update $v_1$ and $w_{11}$ by hand here can overshoot or crawl depending on how big a step is taken.

What this means in practice

The one-step gradient descent update at learning rate $0.1$ moves every parameter a small amount opposite its gradient — $v_1 \to 0.5 - 0.1(-0.3977) = 0.5398$, $w_{11} \to 0.3 - 0.1(-0.0846) = 0.3085$, and so on for all nine. Recomputing the forward pass with the updated weights would show the loss has dropped from $0.2397$, confirming the step moved in a genuinely useful direction. Real networks repeat this exact nine-number (or nine-million-number) update, unchanged in kind, thousands of times per training run; nothing about a production-scale network is different in principle from what happened by hand above, only in size.

Practice

1. Using the deltas already computed, find $\partial L / \partial w_{22}$ and check your arithmetic against the value given in the worked example.
2. Suppose $v_2$ had been $+0.3$ instead of $-0.3$. Redo the sign of $\delta_2$ only, using the chain $\delta_2 = \delta_3 v_2 a_2(1-a_2)$, and explain in words what that sign change means for how hidden neuron 2 should move.
3. Why does the finite-difference check use a small perturbation like $0.001$ rather than, say, $1.0$?