Recurrent Neural Networks

A plain feedforward network takes one fixed-size input and produces one output; it has no notion of "before" and "after." Feed it yesterday's return and today's return as two separate inputs and it treats them as unrelated numbers, with no memory that yesterday even happened. But a lot of what matters in markets — momentum building, volatility clustering, a regime shift unfolding — is defined by order: the same return means something different depending on what came before it.

A recurrent neural network reads a sequence like a trader reading a tape: at every new print, it doesn't start from scratch — it updates a running mental summary of "what's going on," folding the new data point into everything it already believed, and that updated summary is what it carries into the next print. The summary is a fixed-size vector, and crucially the same update rule is applied at every step, so a network trained on ten-step sequences can, at least in principle, be run on sequences of any length.

The mechanics, one symbol at a time

Let $x_t$ be the input at time $t$ (say, today's return and volume) and $h_t$ be the hidden state — the running summary — after processing $x_t$. The recurrence is:

$$h_t = \varphi(W_h h_{t-1} + W_x x_t + b)$$

In words: the new summary is a weighted combination of the previous summary and the current input, squashed through a nonlinearity $\varphi$ exactly like an ordinary neural network layer. $W_h$ says how much of the old summary to carry forward and how, $W_x$ says how to fold in the new data, and $b$ is a baseline. The same three parameters $W_h, W_x, b$ are reused at every single timestep — this weight sharing across time is the entire point, it's what lets one trained network handle sequences of any length rather than needing a separate set of weights per position.

A prediction, if you want one at every step (or just the last one), reads off the current hidden state through an ordinary output layer: $\hat{y}_t = W_y h_t + c$.

Worked example 1: unrolling three steps by hand

A toy RNN with scalar (1-number) state for clarity: $h_t = \tanh(0.6\, h_{t-1} + 0.9\, x_t)$, starting at $h_0 = 0$. Inputs are three days of standardized returns: $x_1 = 0.5$, $x_2 = -0.3$, $x_3 = 0.8$.

Step 1: $h_1 = \tanh(0.6(0) + 0.9(0.5)) = \tanh(0.45) \approx 0.42$.

Step 2: $h_2 = \tanh(0.6(0.42) + 0.9(-0.3)) = \tanh(0.252 - 0.27) = \tanh(-0.018) \approx -0.018$.

Step 3: $h_3 = \tanh(0.6(-0.018) + 0.9(0.8)) = \tanh(-0.0108 + 0.72) = \tanh(0.709) \approx 0.61$.

Notice $h_2$ almost entirely wiped out the memory of the strong up-day at $t=1$ — it landed near zero because the down-day at $t=2$ roughly cancelled it. $h_3$ then builds fresh off that near-zero state. This is a small, concrete instance of what "vanishing memory" looks like: information from early steps can get diluted to near nothing by the time a few more steps have passed, purely from how the weights happen to combine.

Worked example 2: why training is a chain rule down a whole sequence

Training an RNN means backpropagating the error at the final step, $t=3$, all the way back to the weights used at $t=1$ — a technique called backpropagation through time. Using the chain rule, the gradient of the loss at $t=3$ with respect to $h_1$ picks up a factor from every step in between:

$$\frac{\partial h_3}{\partial h_1} = \frac{\partial h_3}{\partial h_2} \cdot \frac{\partial h_2}{\partial h_1}$$

Each factor $\frac{\partial h_t}{\partial h_{t-1}}$ involves $W_h$ multiplied by the derivative of $\tanh$, which is at most 1 and typically much smaller once the state is away from zero. If each factor is, say, 0.3, then over just 10 steps the compounded gradient is $0.3^{10} \approx 0.0000059$ — essentially zero. A signal from an error 10 steps in the future has almost no way to influence weights used 10 steps ago. This is the vanishing gradient problem in concrete numbers, and it's exactly why plain RNNs struggle to learn dependencies spanning more than a handful of steps, and why gated variants (LSTM, GRU) were built specifically to give the gradient a more direct path back.

Use the explorer above to build intuition for a related idea: how a running quantity (there, a sample average; here, the hidden state) settles as more data arrives — and how sensitive that settling process is to how strongly each new point is weighted against the accumulated history.

What this means in practice

Plain ("vanilla") RNNs are rarely used directly today; gated variants — LSTM and GRU — add learned gates that let the network decide what to keep, forget, and output at each step, which mitigates (not eliminates) the vanishing gradient problem and lets them handle dependencies spanning tens to a few hundred steps. In quant work, RNNs and their gated cousins show up for order-book sequence modeling, intraday volatility forecasting, and any setting where the data truly arrives as an ordered stream rather than a fixed table. They've lost ground to attention-based models on long sequences precisely because attention gives every past step a direct connection to the present, sidestepping the step-by-step information bottleneck described above — but RNNs remain cheaper, simpler, and often good enough for shorter, streaming, low-latency settings.