Skip to content
Dreamers Inc.
Education

Attention, Cross-Attention & TransformersAttention is a learned routing rule for context.

Each token asks what matters to it, every other token advertises what it holds, and the model learns how much information should flow between them. Pick a word below and watch it decide where to look.

Try the lens

Try it

Self-attention, one word at a time

Transformers update every token with context from the whole sentence, in parallel and many times over. Choose a word on the left; the strands show where it looks, and the more strands, the more weight.

Attention head

Links a pronoun to the noun it refers to.

Last word of the sentence

Swap it and watch where “it” looks. Google used this pair in 2017 to show what attention learns.

it attends most to animal 64%

  1. The1%
  2. animal64%
  3. didn’t1%
  4. cross1%
  5. the1%
  6. street10%
  7. because1%
  8. it8%
  9. was1%
  10. too1%
  11. tired9%
  12. .1%

Put attention to work on your data 

The mechanism

Query, key and value, explained

Each token starts as an embedding: a vector that stands for the token before it has absorbed any context. Attention refines those embeddings, and to do it each token is projected three ways.

  • Query

    What this token is looking for.

  • Key

    What this token offers to others.

  • Value

    The information it passes along if it is chosen.

Attention(Q, K, V) = softmax(QKᵀ / √dₖ) V

  1. Score QKᵀ

    Compare every query with every key using dot products.

  2. Scale and normalize softmax( · / √dₖ )

    Divide by the square root of the key size, then turn scores into weights that sum to one.

  3. Mix · V

    Each token takes a weighted sum of the values: find what matters, normalize it, move the right information.

Why divide by √dₖ

A query and a key are vectors of length dₖ. When their entries have unit variance, their dot product has variance dₖ, so a 64-wide head produces scores eight times wider than a 1-wide one, and softmax hands nearly all the weight to a single key.

Dividing by √dₖ brings every head back to unit variance. The weights stay soft enough for the model to keep learning which keys matter.

0q · kdₖ = 1dₖ = 16dₖ = 640q · k ⁄ √dₖevery dₖ

Train your team on transformers 

History

Before transformers, and why attention won

Before the transformer, sequence-to-sequence modeling ran mostly on recurrent networks such as LSTMs and GRUs. They worked, but recurrence squeezes everything the model knows through one hidden state, one step at a time, which makes long-range dependencies hard and training slow to parallelize.

Attention links every position to every other in a single step, and it runs beautifully in parallel on GPUs. That let the field push scale far past recurrent systems, and the rest of the stack reorganized around it.

RecurrentAttention

Language models

Self-attention, masking and next-token prediction

When queries, keys and values all come from the same sequence, you get self-attention. A language model masks it so each position may read the past and never the future. In the grid, each row is a query and its hatched cells are the keys it may read.

The lens above is an encoder, so it has no mask: “it” can see “tired”. A decoder that writes one word at a time reads everything it has written so far, all at once, and never looks ahead.

keys →queries →

Translation

Cross-attention and why translation is so central

In cross-attention the queries come from one sequence and the keys and values from another. In the original transformer, the decoder asked which words of the source sentence mattered while it wrote the target sentence: align what is being produced now with the most relevant part of a different sequence.

The paper that made transformers unavoidable was a translation paper. The same mechanism anchors speech translation, transcription and multimodal models, anywhere one stream has to stay tied to another.

In the grid, French rows attend to English columns, and a bigger dot means more weight. Most of it sits on the diagonal, because the two languages mostly share word order. The gold block runs the other way: “European Economic Area” becomes “zone économique européenne”.

TheagreementontheEuropeanEconomicAreawassignedinAugust1992.
L’
accord
sur
la
zone
économique
européenne
a
été
signé
en
août
1992
.

Build translation into your product 

Many views

Multi-head attention and why the model uses many views

One attention pattern expresses one kind of relevance. Multi-head attention runs many in parallel, so a layer can track syntax, coreference, long-range dependencies and position at once, plus behaviors that are harder to name. The head selector in the lens shows three of them.

Honest limits

What attention maps don’t tell you

An attention map shows where one head routes information. A full transformer adds many layers, heads, residual paths and feed-forward blocks, so explaining a model’s behavior takes interventions, ablations and evaluation on top of the maps. Some heads read cleanly, and that is where the work starts.

Test what your model attends to 

Recommended

Visual explainers worth your time

FAQ

Questions people ask

What is attention in a transformer?

Attention is a learned routing mechanism for context. Each token creates a query, key, and value. The model compares queries to keys, converts those scores into weights with softmax, and uses the weights to mix value information into the current representation. Multi-head attention repeats this in parallel so different heads can track different relationships, such as syntax, reference, position, or copying. In plain language: each token asks what matters nearby, then updates itself using the answers.

What is the difference between self-attention and cross-attention?

In self-attention, a sequence attends to itself, so words in one sentence can update each other based on context. In cross-attention, one sequence asks questions of another sequence. That is why translation is such a natural origin story: the decoder can generate a target sentence while attending to the relevant parts of the source sentence. The same idea shows up in image captioning, multimodal models, retrieval-augmented generation, and systems where one representation has to condition on another.

Why did transformers replace many recurrent sequence models?

Transformers made it easier to model long-range context while training in parallel on modern hardware. Recurrent models like LSTMs and GRUs process sequences step by step, which can make long dependencies and large-scale training harder. Transformers also separated sequence position from recurrence, using positional information plus attention instead of a strict left-to-right hidden state. They are not magic, and attention has scaling costs, but the architecture scaled much more cleanly across language, vision, audio, and multimodal systems.

Does an attention map explain what a model is thinking?

Not perfectly. Attention patterns show where information is being routed, which is only part of how a model reasons. A transformer has many layers, heads, residual paths, feed-forward blocks, and learned representations. Some heads are interpretable; others are not usefully human-readable. Attention is a useful window into the machinery; serious interpretation also needs interventions, ablations, and evaluation.

Why divide by the square root of d_k?

Dot products grow with the size of the vectors. If the components of a query and a key are independent with mean 0 and variance 1, their dot product has variance d_k, so scores get large and the softmax saturates: nearly all the weight lands on one token and the gradients become tiny. Dividing by the square root of d_k brings the scores back to unit variance and keeps training stable. The original transformer paper gives exactly this reason.

More reading

Sources
  1. Vaswani et al., Attention Is All You Need (2017)
  2. Bahdanau, Cho and Bengio, Neural Machine Translation by Jointly Learning to Align and Translate (2014)
  3. Google Research: Transformer, a novel neural network architecture for language understanding (Aug 31, 2017)
  4. Grant Sanderson, Neural Networks: Transformers, chapter 6
  5. Stefan Schneider, Understanding Transformers and Attention
  6. hkproj, Transformer From Scratch Notes Diagrams