Attention — how a model uses the context

Every token looks back at the others

Start with a sentence the model has to make sense of — "The cat sat on the mat because it was tired."

What does it point to — the cat, or the mat? You resolved that instantly: the cat. The model has to recover the same link, and it has nothing to work with but the earlier tokens. So to decide what comes next, each position looks back over the tokens before it and gives each one a weight — how much it matters right here. "It" puts most of its weight on "cat", little on "mat" or "the". That weighting is attention.It runs at every generation step: the new token's query weighs all the earlier tokens — which is why each step's cost grows with how much context there is (see the KV cache below).

Two things hold throughout. Attention is causal — a token may look only at tokens before it, never ahead. And the weights are not hand-written rules: the model can reach any earlier token directly, however far back, and learns for itself which ones matter. Think of it as a soft search — every earlier token is something you can look up, you score how relevant each is, and you pull a blend from the best matches rather than a single hit. The widget makes the causal look-back concrete: click any token to see which earlier tokens it leans on; tokens after it grey out, because the model cannot see them yet.

Where the weights come from

Those weights do not come from nowhere. The mechanism behind them borrows its three names — query, key, and value — from an ordinary dictionary lookup: you hold a query, match it against the available keys, and pull back the matching value. Attention does the same thing, softly — it matches against every key at once and blends the values.

First, every token becomes numbers. Before any of this, each token (Part 2) is turned into a list of numbers called its embedding — a learned representation of the token's meaning, arranged so that tokens used in similar ways get similar lists. Everything attention does is arithmetic on these vectors.Real embeddings have hundreds or thousands of dimensions. The widget below uses just two, so the arithmetic fits on the page.

From each embedding, the model makes three vectors. It multiplies the embedding by three separate learned matrices to get a query (what this token is looking for), a key (what it offers), and a value (what it contributes if chosen). Why not just use the embedding directly? Because one vector cannot play three roles at once. The projections let a token's question differ from what it advertises — so the pronoun "it" can learn to look for an animal-like noun such as "cat", even though "it" and "cat" do not mean the same thing. A single shared vector could only ever match whatever is most like itself.

The three matrices are not written by hand. They are part of the model's weights, set by the same training Part 1 described: start from random numbers and nudge them, over enormous amounts of text, toward whatever makes the next-token guess better. What should attend to what is learned, not specified.

Then: compare, normalise, blend. The match score between a token's query and an earlier token's key is their dot product — multiply the matching slots and add them up; it comes out large when the two vectors point the same way. softmax turns the raw scores into weights that add up to 1. And the token's output is the weighted blend of the values — mostly the value of whatever it attended to most. That blend, computed once per token, is a single attention head.

A model runs many heads in parallel — multi-head attention — and each can specialise: one head might track a pronoun to its antecedent, another a verb to its subject, another long-range topic. Their blends are combined, and the whole arrangement is stacked layer upon layer, each re-attending over the one below.

Step back and the shape is familiar: attention is a soft, learned lookup — closer to a content-addressable memory than to a fixed rule. The instinct that it resembles a semantic similarity search is right, with three twists that make it attention: it compares learned projections of the tokens rather than the raw tokens, so relevance is learned and directional; it returns a blend of values rather than one match; and it is causal and multi-headed, looking only backward and through several lenses at once.

The KV cache — why generation stays fast, and what grows

Generation happens one token at a time, and each new token attends back over the whole prefix. Picture writing a sentence word by word: at every new word you glance back over everything written so far. But the earlier words have not changed — re-deriving your notes on them at every step would be wasted effort.

What changes each step, and what doesn't. The only genuinely new thing at each step is the query of the token being generated. Every earlier token's key and value were computed from that token at its fixed position — so for a given prompt they are identical at every step. So cache them. Each new token computes just its own query and attends to the stored keys and values instead of recomputing the prefix. That store is the KV cache, and it is what grows as the context gets longer.

What this explains

With the mechanism in hand, the practical consequences follow. Long context costs memory and speed because the KV cache grows linearly and attention work grows quadratically. A detail buried mid-context can be under-attended — empirical studies call this "lost in the middle" — so a large window does not guarantee uniform use of everything in it. Position and order matter because attention is computed over positions: the same words in a different order produce different weights.

Reasoning, tool use, structured output, and instruction following all ride on this mechanism. Each capability in Part 5 is the same next-token engine shaped by how attention allocates weight across the prompt — which is why the prompting choices that seem cosmetic often have measurable effects.