Part 3 of 5
Context — the model's working memory
One finite token budget — everything the model can see must share it.
The context window is a fixed token budget
Every time a model generates a response it can only attend to the tokens currently in its context window. This window holds everything relevant to the current turn: the system prompt, tool specifications, any files or retrieved passages supplied by the application, and the full conversation history so far. Prompt (input) and completion (output) both draw from the same budget — there is no separate pool for one or the other.Prompt and completion share the same budget. A model advertised as "128 k context" can process at most 128,000 tokens across input and output combined. If the prompt is 120,000 tokens the model has at most 8,000 tokens of headroom for its reply.
The widget below makes the budget concrete. Pick a window size, add files, and send turns until the bar fills. Once the fixed costs (system prompt, tool specs, files) plus history exceed the cap, the oldest history turns are evicted — they simply fall out of the window and the model cannot attend to them.
What competes for the budget
A single context window is divided, without walls, among several claimants. The system prompt carries the developer's standing instructions — persona, rules, tool descriptions, output format. Tool specifications enumerate every function the model is allowed to call, including their parameter schemas; a richly-tooled application can spend several hundred tokens here before the user says anything. Retrieved files — documents, code snippets, memory passages fetched by the application — arrive next and can be large. Finally, the growing conversation history accumulates turn by turn.
When the sum of all these exceeds the window, something must be dropped. Almost always it is the oldest conversation turns, since those are the most likely to be redundant. Choosing what to keep and what to drop is context management — a design problem for whoever builds the application. A well-managed context keeps the most decision-relevant material in the window; a poorly managed one evicts context the model still needs.
Bigger windows aren't free
A larger window gives the model more room and reduces eviction pressure, but capacity has a price — memory and speed both suffer as the window grows. The mechanism behind that cost — attention and the key-value cache — is the next part.
Even when the window is large enough to hold everything, attention is not uniform. Empirical studies find that models attend more reliably to material near the beginning and end of the context, and can miss a detail buried in the middle — an effect sometimes called the "lost in the middle" problem. A bigger budget does not guarantee that everything in it is used equally.