Decoded: Attention Is All You Need, the paper that introduced the Transformer

Before you start

Open only what you need; each one is a short refresher.

Dot products as similarity

The dot product of two vectors multiplies them component by component and adds the results: q·k = Σ q_ik_i. It equals |q||k|cos θ: for fixed vector lengths the score is largest when the vectors align, zero when they are perpendicular and negative when their angle exceeds 90 degrees; changing either length also changes the score. The paper uses it as the compatibility function that scores how well a query matches a key.

Softmax, and what saturation means

Softmax turns a list of scores into positive weights that sum to 1: w_i = e^s_i / Σ_j e^s_j. When the scores are spread far apart, almost all the weight goes to the largest one. When one weight is near 1, the derivatives of softmax with respect to its input scores are small, so the output barely moves when a score changes. The authors suggest this as the reason large unscaled dot products can make learning difficult; the paper speaks of regions with “extremely small gradients”.

Matrix shapes: every query against every key

For self-attention over n positions, stacking the query vectors (length d_k) gives a matrix Q of shape n × d_k and stacking the keys gives K, so QK^T is n × n: entry (i, j) is the dot product of query i with key j. More generally n_q queries and n_k keys give n_q × n_k scores; V has n_k rows and the output has n_q rows of width d_v. One matrix multiply therefore scores every pair of positions at once, which is why attention parallelises well on hardware built for matrix multiplication. The square table is the self-attention case; encoder–decoder attention can have different numbers of query and key positions (section 6).

Embeddings, and one table used twice

An embedding is a learned table with one row of length d_model per token in the vocabulary; looking a token up returns its row. The paper shares one weight matrix between the input embedding, the output embedding and the linear layer before the final softmax, and multiplies the embedding weights by √d_model (paper §3.4).

Residual connections and layer normalisation

A residual connection carries a sub-layer's input x unchanged around the sub-layer and adds it to Sublayer(x) before layer normalisation; the normalised result is a new vector, not a copy of x. Layer normalisation (the paper's reference [1]) then rescales the sum to mean 0 and standard deviation 1 across its d_model values, followed by a learned gain and bias. Every sub-layer in the Transformer produces LayerNorm(x + Sublayer(x)).

1. The encoder–decoder stack (paper §3 and §3.1)

Earlier translation systems commonly used recurrent encoder–decoder networks with attention on top (§1–§2): recurrence makes every position wait for a chain of hidden-state computations, which limits how much of a sentence can be processed in parallel. This paper replaces that sequence-aligned recurrence with attention-based layers; attention itself predates the paper (§2). Each input token first becomes an embedding, a learned vector of width d_model, to which a positional encoding is added (§3.4–§3.5) before it enters the stack.

The Transformer keeps the usual shape of a translation model: an encoder maps the source sentence (x₁, …, x_n) to a sequence of continuous vectors z, and a decoder generates the output one token at a time, feeding back the tokens it has already produced. This is called auto-regressive generation. What is new is that neither half contains a recurrent or convolutional layer.

Unless stated otherwise, the sizes in this lesson are the base model’s: d_model = 512, d_ff = 2048, h = 8 (the big model uses 1024, 4096 and 16); the figures label their own smaller illustrative sizes.

Each half is a stack of N = 6 layers with the same structure and separate learned parameters (“identical” describes the structure, not shared weights). During generation the encoder computes z once for the source sentence and the decoder predicts successive output tokens from the prefix generated so far, reading z in every one of its layers; during training the known reference translation is shifted right and all target positions are processed together, and the causal mask stops a position from using later shifted inputs; shifting and masking together ensure that the prediction of output y_i uses only earlier output tokens. The decoder's input is the output sentence shifted right by one position, so the prediction at position i can only use outputs before i.

2. Inside one layer (paper §3.1)

An encoder layer has two sub-layers: multi-head self-attention, then a position-wise feed-forward network. A decoder layer adds a third sub-layer, encoder–decoder attention over the encoder's output, placed between its masked self-attention and its feed-forward network (so it is second in execution order). Every sub-layer is wrapped as LayerNorm(x + Sublayer(x)). To make the residual addition possible, every sub-layer and the embeddings produce vectors of the same width, d_model = 512.

3. Scaled dot-product attention (paper §3.2.1)

An attention function maps a query and a set of key–value pairs to an output: a weighted sum of the values, where each weight comes from how well the query matches that value's key. The paper's version packs all queries, keys and values into matrices and computes

Attention(Q, K, V) = softmax(QK^T / √d_k) V (eq. 1)

Each row of Q is a query vector, each row of K is a key vector and the matching row of V is the value that can be retrieved; all three are learned projections of the sequence representations entering the layer, and the scores decide how much of each value is mixed into the output. In general Q is n_q × d_k, K is n_s × d_k and V is n_s × d_v: the scores and weights are n_q × n_s, the output is n_q × d_v, and d_v need not equal d_k. The only difference from plain dot-product attention is the factor 1/√d_k. Compared with additive attention, which scores pairs with a small feed-forward network, the dot-product form is much faster in practice because it is a matrix multiply.

4. Why divide by √d_k (paper §3.2.1, footnote 4)

The paper observes that without scaling, dot-product attention falls behind additive attention for large d_k. Its explanation is in footnote 4: if the components of q and k are independent with mean 0 and variance 1, then q·k has mean 0 and variance d_k. Under those assumptions d_k = 64 gives an unscaled score a standard deviation of 8; the authors suspect that such large score differences push the softmax to put nearly all its weight on one key, where its gradients are very small (a hypothesis in the paper, not a guarantee for any particular row). Dividing by √d_k brings the variance back to 1 for any d_k.

5. Multi-head attention (paper §3.2.2)

A single attention is one weighted average per position, which can blend information that would be more useful kept separate (§2). Multi-head attention runs several such averages at once. Instead of one attention over d_model-wide vectors, the paper projects the queries, keys and values h times with different learned matrices W_i^Q, W_i^K, W_i^V, runs attention on each projection in parallel, concatenates the h outputs and projects once more with W^O. The paper's reason: “With a single attention head, averaging inhibits this”, where “this” is attending to different kinds of information at different positions at the same time.

The first figure follows one head's projection cell by cell; the second runs the heads in parallel. Each head projects the full d_model-wide input into its own learned subspace: the narrower widths divide the output budget, not the input into fixed slices, and the coloured split in the figure is only one illustrative projection.

The base model uses h = 8 heads with d_k = d_v = d_model/h = 64, so the total cost is close to one full-width head.

6. Three uses of attention, and the mask (paper §3.2.3)

The same multi-head attention appears in three places. In encoder self-attention, queries, keys and values all come from the previous encoder layer, and every position can see every other. In decoder self-attention, they come from the previous decoder layer, but query row i may read keys at positions j ≤ i only: information at a future position cannot influence that row's prediction. In encoder–decoder attention, Q is projected from that decoder layer's masked self-attention output (after its residual addition and normalisation) and K and V from the final encoder output z, so every output position can look at the whole source sentence.

The decoder's restriction is implemented inside the attention itself: before the softmax, every score that would let a position look ahead is set to −∞, which makes its weight exactly 0.

7. The position-wise feed-forward network (paper §3.3)

Each layer also contains a small two-layer network applied to every position separately and identically: FFN(x) = max(0, xW₁ + b₁)W₂ + b₂ (eq. 2). The input and output are d_model = 512 wide and the hidden layer is d_ff = 2048 wide. The weights are shared across positions within a layer and differ from layer to layer, which the paper notes is the same as two convolutions with kernel size 1.

8. Positional encoding (paper §3.5)

Unmasked self-attention without positional information is permutation-equivariant: permuting the input positions permutes the outputs the same way. The paper therefore adds a positional encoding of width d_model to the token embeddings at the bottom of both the encoder and the decoder stack, built from sines and cosines:

PE(pos, 2i) = sin(pos / 10000^2i/d_model), PE(pos, 2i+1) = cos(pos / 10000^2i/d_model)

Each dimension is a sinusoid, and the sine and cosine in a pair share one frequency; and the wavelength of pair i is 2π·10000^2i/d_model, a geometric progression that starts at 2π and approaches 10000·2π (with d_model = 512 the largest wavelength used is about 60,611). The authors chose this form because, for any fixed offset k, PE(pos+k) is a linear function of PE(pos), which they hypothesised would make relative positions easy to learn. Learned position embeddings gave nearly identical results (Table 3, row E); the authors kept the sinusoids because they might extrapolate to sequences longer than those seen in training, a motivation rather than a demonstrated guarantee.

9. Why self-attention (paper §4 and Table 1)

The paper compares layer types on three measures: computation per layer, the number of operations that must run one after another, and the longest path a signal travels between two positions. A self-attention layer costs O(n²·d) but needs only O(1) sequential steps and connects any two positions directly. A recurrent layer costs O(n·d²) and needs O(n) sequential steps. Comparing these leading-order expressions favours self-attention when n < d, a regime the paper identifies as common in sentence-level translation; actual runtime also depends on constants and implementation. These are the asymptotic costs of the sequence-mixing operation in Table 1, not whole-model runtimes: the learned projections and the feed-forward network add work of order n·d², and stacked layers and auto-regressive generation keep their sequential dependencies. For very long sequences the paper proposes, as future work, restricting attention to a neighbourhood of r positions, which reduces the term to O(r·n·d) and raises the maximum path length to O(n/r); the reported models use full attention.

10. Training: the warm-up schedule (paper §5)

At each training position the decoder predicts the next reference token from the source and the earlier reference tokens; the probability it assigns to the reference token drives the weight updates. The experiments use the WMT 2014 translation data with sub-word tokens and batches grouped by approximate sentence length (§5.1). The models were trained with the Adam optimiser (β₁ = 0.9, β₂ = 0.98, ε = 10⁻⁹) on one machine with 8 NVIDIA P100 GPUs: 100,000 steps (about 12 hours) for the base model and 300,000 steps (3.5 days) for the big model. The learning rate follows eq. 3, lrate = d_model^−0.5 · min(step^−0.5, step · warmup_steps^−1.5): it rises linearly for the first warmup_steps = 4000 steps and then decays with the inverse square root of the step number. Two regularisers act during training: dropout (P_drop = 0.1 for the base model) randomly zeros components of each sub-layer's output before the residual addition and normalisation, and of the embedding-plus-position sums (the layer figures omit it); label smoothing (ε_ls = 0.1) moves some target probability from the reference token to the alternatives, which the paper reports hurts perplexity but improves accuracy and BLEU.

11. What the paper measured (paper §6.1 and Table 2)

On WMT 2014 English-to-German, the big Transformer reaches 28.4 BLEU, more than 2 BLEU above every previously reported model including ensembles, and even the base model (27.3) beats them at a fraction of the training cost. On English-to-French, the big model reaches 41.8 BLEU in Table 2 and the abstract (the §6.1 text says 41.0), a new single-model state of the art at under a quarter of the previous best model's training cost. BLEU is a standard score for how closely a machine translation matches human reference translations; higher is better. These test results come from one model whose weights are the average of recent checkpoints (five for base, twenty for big) decoded with beam search (beam 4, length penalty α = 0.6); this is not an ensemble of separately trained models. The training FLOPs are estimates from training time, GPU count and an assumed sustained throughput per GPU.

12. How many heads? (paper §6.2 and Table 3)

To see which parts matter, the authors varied the base model and measured English-to-German quality on the development set. Rows (A) of Table 3 change the number of heads while keeping the computation constant: a single head is 0.9 BLEU worse than the best setting, and quality also drops off with too many heads. Other rows show that a smaller d_k hurts, that bigger models are better and that dropout helps avoid over-fitting. These development-set runs use beam search without checkpoint averaging. The table also reports perplexity: the exponential of the mean negative log probability given to the reference next word piece, so lower is better; it scores probabilities on a known reference, whereas BLEU scores a decoded translation, and the two need not move together. Values are per word piece and are not comparable with per-word perplexities.

What the paper showed (paper §7)

The paper presents the Transformer as the first sequence transduction model based entirely on attention, replacing the recurrent layers of encoder–decoder models with multi-headed self-attention. “Entirely on attention” means that attention replaces recurrent or convolutional mixing between positions; embeddings, the position-wise feed-forward networks, residual connections, normalisation and the output softmax remain. On the studied translation tasks it trained substantially faster than the compared recurrent and convolutional systems and set a new state of the art on both WMT 2014 tasks; on English-to-German the best model outperforms all previously reported ensembles. For English constituency parsing (§6.3, Table 4), a four-layer Transformer with d_model = 1024 scores 91.3 F1 trained on about 40,000 Wall Street Journal sentences and 92.7 F1 in the semi-supervised setting with about 17 million training sentences; these are parsing scores, not BLEU, and they show transfer beyond translation without beating every parser. The appendix visualises attention patterns consistent with long-distance dependencies, anaphora and differing head behaviour, as suggestive examples from particular layers and heads. The authors close by planning to apply attention to images, audio and video, to study restricted local attention for large inputs, and to make generation less sequential.

Sources and credits

Paper — Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser and Illia Polosukhin. Attention Is All You Need. Advances in Neural Information Processing Systems 30 (NIPS 2017), 2017. arXiv:1706.03762
Authors' code — The paper names tensor2tensor as the code its authors used to train and evaluate the models. This lesson does not use or reproduce it.
Figures — All figures on this page are original drawings written for this lesson from the paper's text, with small illustrative sizes (4 tokens, d_model = 8, d_k = 4) where the paper's real sizes would not fit on screen. Nothing is reproduced from the paper's figures; numbers quoted from its tables are cited to the table.

The explanations and interactive drawings are original interpretations of the cited research. Rights in the papers remain with their respective rightsholders; citation does not imply endorsement.