Tag: transformers-attention
Concepts
- ALiBi and Relative Position Bias
- Causal and Padding Attention Masks
- Cross-Attention Between Encoder and Decoder
- Feedforward Blocks and Gated Linear Units
- Learned Positional Embeddings
- Multi-Head Attention
- Pre-Norm vs Post-Norm Transformer Blocks
- Queries, Keys and Values
- Scaled Dot-Product Attention and the √d Factor
- Sinusoidal Positional Encoding
- The Residual Stream View of Transformers