Transformers and Attention
Transformers and Attention
Key jargon
| Term | Plain-language meaning |
|---|---|
| Transformer | A neural-network architecture built around attention and feed-forward blocks. |
| Attention | A learned mechanism that mixes information from different token positions. |
| Query, key, value | Three learned projections used to calculate what each token should read and what information it receives. |
| Positional information | Signals that let a model distinguish token order. |
Key concepts
- Self-attention lets every token build a context-dependent representation from other permitted tokens.
- Attention weights show information routing inside a calculation; they are not a complete explanation of model reasoning.
Concept map
flowchart LR
A["Token representations"] --> B["Create queries keys values"]
B --> C["Compute attention"]
C --> D["Build contextual representations"]Learning objectives
- Explain attention as context-dependent information routing.
- Distinguish transformer architecture from a complete chat application.
Mental model
For each token representation, attention produces a query, key, and value. Query–key similarity determines how much each value contributes to the next representation. Multiple heads can learn different relationships. Feed-forward layers transform each position; residual connections and normalization stabilize deep networks; positional information supplies ordering.
Transformers made sequence processing highly parallel during training. Modern systems add many variations: decoder-only, encoder-only, encoder–decoder, mixture-of-experts, sparse attention, multimodal adapters, and optimized attention kernels.
Important limits
- Attention weights are not a complete causal explanation.
- A long context window does not mean every token is used equally well.
- The original transformer paper demonstrated machine translation; modern capabilities result from later scaling, data, objectives, and post-training.
Exercise
Take: “The trophy did not fit in the suitcase because it was too large.” Identify which earlier words the representation of “it” should relate to and why context matters.