Orthogonal and Identity Initialization

Xavier and He initialization control the variance of a weight matrix so that, on average across many random draws, signal doesn't systematically grow or shrink. But "on average" isn't a guarantee for any one specific matrix, and in a recurrent network the same weight matrix gets applied over and over — dozens or hundreds of times per sequence — so even a small per-step distortion compounds relentlessly. Orthogonal initialization sidesteps the whole problem: it starts with a matrix that provably preserves the length of any vector it multiplies, exactly, every time.

The analogy: a rotation versus a stretch

Multiplying a vector by an orthogonal matrix is like rotating it in place — the direction can change, but the length cannot, ever, by construction. Multiplying by an ordinary random matrix is more like squeezing a ball of dough: some directions get stretched, others squashed, and if you repeat the squeeze a hundred times in a row (as a recurrent network does across a hundred time steps), the dough ends up wildly distorted — either flattened to almost nothing or stretched to a sliver. A rotation, repeated a hundred times, is still just a rotation: the ball stays the same size.

The idea, in one line of matrix algebra

A square matrix $Q$ is orthogonal if $Q^\top Q = I$, the identity matrix. In words: $Q$'s own transpose is also its inverse. The direct consequence is that for any vector $v$, $\|Qv\| = \|v\|$ — multiplying by $Q$ changes a vector's direction but never its length. Every eigenvalue of an orthogonal matrix has absolute value exactly 1, which is exactly the condition needed to stop repeated multiplication from exploding (eigenvalues $>1$) or vanishing (eigenvalues $<1$). Identity initialization is the simplest special case: start the recurrent weight matrix as $I$ itself (or close to it, plus small noise), which trivially satisfies $Q^\top Q = I$ and starts the network by passing each hidden state straight through unchanged.

Drag the matrix entries below and watch the unit circle deform into an ellipse, with the eigenvectors and determinant displayed — an orthogonal matrix is exactly the special case where the ellipse stays a circle and every eigenvalue sits on the unit circle.

Worked example 1: a 2×2 rotation matrix, checked by hand

Take $Q = \begin{pmatrix} \cos\theta & -\sin\theta \\ \sin\theta & \cos\theta \end{pmatrix}$ with $\theta = 30°$, so $Q = \begin{pmatrix} 0.866 & -0.5 \\ 0.5 & 0.866 \end{pmatrix}$. Apply it to $v = (1, 0)$: $Qv = (0.866, 0.5)$. Its length is $\sqrt{0.866^2 + 0.5^2} = \sqrt{0.75 + 0.25} = \sqrt{1} = 1$ — unchanged from $\|v\| = 1$. Apply $Q$ again, and again — after 100 repeated applications the vector has simply rotated to angle $100 \times 30° = 3000° \equiv 240°$, but its length is still exactly 1. A random non-orthogonal matrix with the same average variance would not offer that guarantee.

Worked example 2: what a non-orthogonal matrix does over many steps

Take $A = \begin{pmatrix} 1.05 & 0 \\ 0 & 0.95 \end{pmatrix}$ — a mild stretch along one axis, mild shrink along the other, with entries of a totally reasonable size. Apply it to $v=(1,1)$ repeatedly: after $n$ steps, the first coordinate is $1.05^n$ and the second is $0.95^n$. At $n = 100$: $1.05^{100} \approx 131.5$ and $0.95^{100} \approx 0.0059$ — one coordinate has exploded roughly 130-fold, the other has vanished to under a percent of its start. A recurrent weight matrix like $A$ (perfectly innocuous per-entry) predictably explodes or vanishes across a long sequence; $Q$ above stays at magnitude exactly 1 after the same 100 steps.

What this means in practice

Orthogonal initialization is the standard choice for the recurrent weight matrices in vanilla RNNs and is available directly in frameworks (nn.init.orthogonal_ in PyTorch); identity initialization for the recurrent weights, paired with ReLU activations, was shown to let simple RNNs learn dependencies hundreds of steps long that would otherwise vanish. Note that orthogonality is a property of square matrices used repeatedly on the same dimensionality — it doesn't directly apply to a feedforward layer changing width from, say, 512 to 128 inputs to outputs, which is why Xavier and He remain the standard choice for ordinary feedforward and convolutional layers.