Backprop Through a Two-Layer Network by Hand

Reading about backpropagation and actually pushing real numbers through a real network by hand are two different levels of understanding, and only the second one leaves you able to spot a bug in someone else's implementation. This page does exactly that: one small network, two inputs, two hidden neurons, one output, every single number written down, forward and backward, with a check at the end that proves the gradients are correct.

The analogy: a two-stage supply chain

A raw material's price moves. To know how much that move affects the price of a finished product, you cannot look at the raw material and the finished product alone, the effect passes through an intermediate processing stage first. If a $1 rise in the raw material adds$2 to the cost of an intermediate part, and a $1 rise in that intermediate part's cost adds$3 to the finished product, then the raw material's total effect on the finished product is $2 \times 3 = 6$ dollars, found by multiplying the effect at each stage. A two-layer network has exactly this structure: an input affects a hidden neuron, and that hidden neuron affects the output, and the input's total effect on the output is the product of the two local effects, chained twice instead of once.

The network

Two inputs $x_1 = 1.0$, $x_2 = 0.5$. Two hidden neurons, each a weighted sum of the inputs passed through a sigmoid, $\sigma(z) = 1/(1+e^{-z})$:

with weights $w_{11}=0.3,\ w_{12}=-0.2,\ b_1=0.1$ and $w_{21}=0.4,\ w_{22}=0.1,\ b_2=-0.05$. A single linear output neuron combines the two hidden activations: $\hat y = v_1 a_1 + v_2 a_2 + c$, with $v_1 = 0.5$, $v_2 = -0.3$, $c = 0.2$. Target $y = 1.0$, loss $L = \frac{1}{2}(\hat y - y)^2$.

Worked example 1: the full forward and backward pass

Forward pass, computed and kept for later:

Backward pass. Start at the output, whose local slope is $\partial L/\partial \hat y = \hat y - y = 0.3076 - 1 = -0.6924$. Because the output layer is linear, that is also $\delta_3 = \partial L/\partial z_3 = -0.6924$.

Output weights, each output weight's gradient is $\delta_3$ times the hidden activation it multiplied:

Into the hidden layer, each hidden unit's incoming gradient is $\delta_3$ times the output weight it fed into, times its own sigmoid's local slope $a(1-a)$:

Notice $\delta_2$ came out positive even though $\delta_3$ is negative, because $v_2$ is negative, the second hidden neuron helps the output by shrinking, and backprop finds that sign flip automatically, exactly as it must.

Hidden weights, each hidden weight's gradient is its unit's $\delta$ times the input it multiplied:

Nine numbers, six weights and three biases, all obtained from one forward pass and one backward sweep, never re-running the network from scratch for each one.

Worked example 2: verifying two gradients by finite difference

Nudge $v_1$ up by $0.001$ to $0.501$, holding everything else fixed, and redo only the parts of the forward pass that change: $\hat y' = 0.501(0.5744) - 0.3(0.5987) + 0.2 = 0.3082$, giving $L' = \frac{1}{2}(1 - 0.3082)^2 = 0.2393$.

That is within rounding of the backprop value $\partial L/\partial v_1 = -0.3977$ computed above, a genuine, independent check, not a restatement of the same calculation. Now nudge $w_{11}$ up by $0.001$ to $0.301$ instead: $z_1' = 0.301$, $a_1' = \sigma(0.301) = 0.5747$, $\hat y' = 0.5(0.5747) - 0.3(0.5987) + 0.2 = 0.3077$, $L' = \frac{1}{2}(1-0.3077)^2 = 0.2396$.

Matching backprop's $\partial L/\partial w_{11} = -0.0846$. Both checks land within finite-difference rounding error of the analytic values, confirming the backward sweep is correct, and this is exactly the gradient-checking routine every hand-rolled implementation should run once before trusting it.

Every one of the nine gradients above feeds into exactly the kind of step the explorer above shows: a small move opposite the gradient's direction, one number at a time. Drag the step size and watch how the same mechanics that update $v_1$ and $w_{11}$ by hand here can overshoot or crawl depending on how big a step is taken.

What this means in practice

The one-step gradient descent update at learning rate $0.1$ moves every parameter a small amount opposite its gradient, $v_1 \to 0.5 - 0.1(-0.3977) = 0.5398$, $w_{11} \to 0.3 - 0.1(-0.0846) = 0.3085$, and so on for all nine. Recomputing the forward pass with the updated weights would show the loss has dropped from $0.2397$, confirming the step moved in a genuinely useful direction. Real networks repeat this exact nine-number (or nine-million-number) update, unchanged in kind, thousands of times per training run; nothing about a production-scale network is different in principle from what happened by hand above, only in size.

Practice

1. Using the deltas already computed, find $\partial L / \partial w_{22}$ and check your arithmetic against the value given in the worked example.
2. Suppose $v_2$ had been $+0.3$ instead of $-0.3$. Redo the sign of $\delta_2$ only, using the chain $\delta_2 = \delta_3 v_2 a_2(1-a_2)$, and explain in words what that sign change means for how hidden neuron 2 should move.
3. Why does the finite-difference check use a small perturbation like $0.001$ rather than, say, $1.0$?