Part 2 of 5
Tokens — how a model reads and counts
Models see tokens, not words. That shapes context limits and cost in ways that are easy to underestimate.
Models read tokens, not words or letters
Before a model processes any text, that text is chopped into tokens by a separate piece of software called a tokenizer. A token is not a word and not a character — it is a piece of text from a fixed vocabulary: sometimes a whole common word, sometimes a word fragment, sometimes a punctuation mark or a space. The model only ever sees the sequence of integer IDs corresponding to those tokens. It has no access to the raw string.
The widget below makes this concrete. Type anything — or pick a preset — and see how the text is carved into pieces. Whole familiar words become one chip; rarer or longer words split into a root and one or more suffixes; digits and punctuation each become their own token. This toy splitter is illustrative rather than real BPE, but the pattern it reveals is the same: short, high-frequency pieces are kept whole; everything else is composed from a smaller set of parts.
Illustrative — a real tokenizer uses a learned vocabulary (BPE). Chip colours: word subword number punct / space
Why this matters to you
Every model has a context limit measured in tokens, not characters. A model advertised as "128 k context" can hold roughly 128,000 tokens in its active window at once. Once that window fills up, earlier content is forgotten — the model cannot attend to it when generating the next token. Cost, for hosted models, is also priced per token (separately for input and output). Getting a feel for token counts — not byte counts, not word counts — is therefore a practical skill.
The rough English rule of thumb is 1 token ≈ ¾ of a word, or about 4 characters. A page of English prose runs around 250–300 tokens. But this rule breaks down quickly outside of plain English, as the next section shows.The chars/token ratio in the widget's status line makes this concrete: plain English prose sits near 4–5 chars/token; Python code often drops to 2–3; JSON with lots of structure and braces can go lower still.
What a real tokenizer does
Code and other languages cost more
Identifiers, indentation, and structured syntax fragment into more tokens than plain English prose. A camelCase function name like getUserName may split into three or four tokens; underscores in snake_case names often become separate tokens; repeated indentation whitespace accumulates token cost that prose does not pay. Switch the widget's preset to "Python code" or "JSON" and watch the chars/token ratio fall — fewer characters per token means more tokens for the same byte count.This is why a code file fills a context window faster than the same byte count of prose. If a conversation is going long with a lot of code pasted in, the context limit is the thing to watch.
Non-Latin scripts carry similar or greater overhead. Many CJK characters are common enough to warrant their own token in large vocabularies, but accented or less-common Unicode falls back to multi-byte sequences — multiple tokens per character. The "Accented text" preset gives a rough sense of this, though our toy tokenizer gives each CJK character its own token rather than consulting a real vocabulary.