The next-token machine

This is the first leg of a three-part trilogy. Here, the concepts: what a language model is, how it generates text, and why it behaves the way it does. The second part, "Running one for real", applies these ideas with real numbers — latency, token counts, cost. The third part is a field reference for practitioners. The arc is learn, apply, recall.

A model predicts the next token

Give a language model a sequence of text — a sentence, a paragraph, the start of a story — and it does one thing: it outputs a probability for every possible next token. Not one answer. Every candidate, all at once, each with a weight. Pick one token, append it to the sequence, feed the whole thing back in, and repeat. That loop is the complete engine. Everything a model does — answering questions, writing code, translating a sentence — is this one operation iterated until the output is done.

The word "token" rather than "word" is deliberate: models don't read characters or words, they read pieces of words according to a learned vocabulary. Part 2 covers the details of tokenisation; for now, think of a token as roughly a short word or a common syllable.

The process of feeding the output back as the next input is called autoregression. There is no separate planning step, no internal scratchpad invisible to the user, no look-ahead. The model commits to one token at a time and then treats what it just wrote as given.

Where the probabilities come from

The probability distribution over next tokens is computed by a large set of numbers called parameters or weights. These numbers are fixed after training: a process in which the model was shown enormous amounts of text and adjusted its weights repeatedly to get better at predicting what came next. Training is slow and expensive. Running the trained model — inference — is fast: it is just arithmetic over the weights.A "small" model (amber) and a "large" model (violet) differ mainly in how many parameters they have — billions vs hundreds of billions. More parameters generally means a better-calibrated guess, not a different mechanism. The loop is identical.

A base model is trained purely to continue text: given what came before, predict what comes next. Ask it a question and it is as likely to continue with another question as with an answer, because that is what questions look like in the training data.

An instruct or chat model starts from a base model and is then post-trained — fine-tuned on examples of helpful responses and shaped by human feedback — to follow instructions and maintain a conversational tone. The next-token mechanism is unchanged; the weights have been adjusted to make "answer helpfully" a likely continuation of "Here is a question: …"

Why it sometimes makes things up

A language model optimises for plausible continuation. Given the text so far, it chooses tokens that form a likely continuation of that text — likely according to patterns absorbed from training data. It does not have a pointer to a fact database, and it does not know which of its continuations are true.

When a model produces confident-sounding text that is factually wrong — sometimes called a "hallucination" — this is the same machinery working as designed, not a bug in the conventional sense. The model found a plausible-sounding next token, and another, and another. The result reads well. It happens to be wrong.

This is worth holding clearly in mind because it shapes every interaction. The model has no internal signal distinguishing "I know this" from "I am generating something plausible here." The signal for catching errors must come from outside — from the user, from grounding in retrieved documents, from checking outputs against known facts.

Turning the dial: temperature

Every token in the model's vocabulary has a raw score called a logit at each generation step. To turn logits into a probability distribution, the model applies a function called softmax. Temperature is a single number that controls how sharply peaked that distribution is before sampling from it.

At low temperature the highest-probability token dominates: the model nearly always picks the same token given the same context. This is called greedy decoding at the limit. The output is predictable and consistent, but can become repetitive. At high temperature the distribution is flatter: lower-probability tokens get more of the weight. The output is more varied, sometimes creative, but also more likely to veer into unlikely continuations. The widget above lets you see this directly — drag temperature down toward 0 and watch one token take nearly all the probability mass; drag it up and watch the distribution spread.