Deep Architectures

Read these first

1. 01Reconstruction Error as an Anomaly Score

   An autoencoder trained only on normal data learns to compress and rebuild normal patterns well. When it meets something abnormal, its reconstruction is bad, and that badness, measured directly, is a usable anomaly score with no labeled anomalies required.

2. 02ResNet and Identity Mappings

   Stacking more layers onto a deep network should only ever help, but past a certain depth it makes training worse. ResNet fixes this with a trick so simple it looks like cheating: let each layer default to changing nothing at all.

3. 03U-Net: Encoder-Decoder With Skip Connections

   A U-Net compresses an input down to its coarse shape and then rebuilds it at full resolution, but rebuilding from the compressed version alone loses fine detail. Skip connections smuggle that detail across directly, from each downsampling step to its matching upsampling step.

4. 04Mixture-of-Experts Layers

   Instead of one huge network processing every input the same way, a mixture-of-experts layer keeps a bank of smaller sub-networks and routes each input to only a few of them, buying the capacity of a much bigger model while paying the compute cost of a much smaller one.

Then the rest

1. 05Bottleneck Blocks and Parameter Efficiency
2. 06Choosing the Autoencoder Bottleneck Dimension
3. 07Deep Sets and Permutation-Invariant Networks
4. 08Denoising Autoencoders
5. 09FT-Transformer for Tabular Data
6. 10Network Pruning and Structured Sparsity
7. 11One-Dimensional Convolutions for Price Series
8. 12Receptive Fields and Dilation
9. 13Siamese and Twin-Tower Networks
10. 14TabNet: Attentive Tabular Learning
11. 15Two-Tower Retrieval Architectures
12. 16Wide vs Deep Network Tradeoffs
13. 17Depthwise Separable Convolutions
14. 18Neural Ordinary Differential Equations
15. 19Skip-Connection Topologies and Gradient Flow
16. 20Sparse Autoencoders
17. 21Squeeze-and-Excitation Blocks
18. 22Top-K Gating and Sparse Routing