Sequence Models

Read these first

1. 01Backpropagation Through Time

   A recurrent network reuses the same weights at every timestep, so training it means adding up how much each timestep's error depends on those shared weights, chained all the way back through the sequence. That chaining is backpropagation through time, and it is also where recurrent networks quietly fail.

2. 02LSTM Gate Mechanics: Forget, Input, Output

   An LSTM fixes the vanishing-gradient problem of plain recurrent networks by keeping a separate memory lane, the cell state, that information can travel down almost unchanged. Three learned gates decide what joins that lane, what leaves it, and what gets read from it.

3. 03Sequence-to-Sequence Encoder-Decoder Models

   Translating a sentence, or forecasting a multi-day price path from a multi-day history, both need to map one sequence to another sequence of a different, possibly unequal length. Encoder-decoder models solve this by compressing the input sequence into a single summary, then generating the output sequence from that summary one step at a time.

4. 04Mamba and Selective State Spaces

   S4's linear state-space layers process every sequence with the same fixed dynamics regardless of content. Mamba makes those dynamics depend on the current input itself, letting the model choose what to remember and what to skip, while keeping the linear-time efficiency that made S4 attractive over attention.

5. 05State-Space Models and S4

   S4 revived a decades-old control-theory idea, the linear state-space model, and showed it can be trained as a neural network layer that processes very long sequences without the quadratic cost of attention or the vanishing gradients of a recurrent network.

Then the rest

1. 06Sliding-Window Supervision for Time Series
2. 07Bahdanau and Luong Attention
3. 08Beam Search Decoding
4. 09Bidirectional Recurrent Networks
5. 10Causal Padding and Look-Ahead Leakage
6. 11DeepAR Probabilistic Forecasting
7. 12Echo State Networks and Reservoir Computing
8. 13GRU vs LSTM Tradeoffs
9. 14Handling Irregular and Missing Timestamps
10. 15N-BEATS Interpretable Forecasting
11. 16Recurrent and Variational Dropout
12. 17Recursive vs Direct Multi-Horizon Forecasting
13. 18Stacked and Deep Recurrent Networks
14. 19Teacher Forcing and Exposure Bias
15. 20Truncated BPTT and Window Length
16. 21WaveNet and Dilated Causal Convolutions
17. 22Neural Hawkes Processes for Event Sequences