Tokenizer & Context Window
A model cannot read your letters. It reads tokens — small reusable pieces of text, each one a number. Type anything and watch your words snap into the exact pieces a model sees, then fill a context window.
🧱 The one-line intuition
Think of a LEGO set. You can't hand a model a finished castle — you hand it bricks. Tokenization snaps your text into standard bricks the model already owns. Common words are big bricks ( the); rare words are broken into smaller ones (token + ization). The model only ever sees the brick numbers — never your letters. This matters because you pay per brick, the model can only hold so many bricks at once, and the way text gets chopped quietly changes cost, speed, and even accuracy.
Tokens the model sees
13 tokens · 1.30/word· = space, ⏎ = newline, ⇥ = tab · Each colored chip is one token — the number below it (toggle “Show IDs”) is the ID the model actually reads.
Tokens
13
Characters
72
Words
10
Chars / token
5.54
Context window usage — GPT-4.1 (1M)
13 / 1,047,576This text fills <0.01% of a 1,047,576-token window. A window this large fits a whole book — most everyday prompts use only a sliver of it.
Same text, different models
The exact same text turns into a different number of tokens depending on the model. Newer models pack text more efficiently, so more fits in the window.
| Model | Context | Tokens | % of context |
|---|---|---|---|
| GPT-4.11M | 1,047,576 | 13 | <0.01% |
| GPT-4.1 mini1M | 1,047,576 | 13 | <0.01% |
| GPT-4o | 128,000 | 13 | 0.010% |
| GPT-5 | 400,000 | 13 | <0.01% |
| o3 (reasoning) | 200,000 | 13 | <0.01% |
| GPT-4 Turbo | 128,000 | 13 | 0.010% |
| GPT-3.5 Turbo | 16,385 | 13 | 0.079% |
Step by step
How your text becomes tokens
You type text
To you it is words and sentences. To the model it is nothing yet — a model cannot read letters directly. It first has to be turned into pieces the model already knows.
Text becomes bytes
Every character is raw bytes underneath. An English letter is one byte; an emoji like 😀 is four. The model's true alphabet is bytes — which is why anything, in any language, can be tokenized.
Bytes merge into tokens
The tokenizer learned which pieces appear together constantly and glues them into reusable bricks. Common words survive as one token; rare words stay split. Frequency decides the brick size.
Tokens become ID numbers
Each brick is looked up in a fixed dictionary (usually 32K–128K entries) and replaced by an integer. The model does math on these numbers — your original text is gone.
IDs fill the context window
The IDs line up in a row. The model can only hold so many at once — that budget is the context window. It counts the instructions, the history, your question, and the answer being written.
Where the tokens come from
How the vocabulary is built (BPE)
Above, you saw the finished tokens. But where does that list of “bricks” come from? A tokenizer is trained: it starts with single characters and repeatedly glues the most frequent adjacent pair into a new token — a process called Byte-Pair Encoding. Type below and watch a vocabulary form, one merge at a time.
Blue chips are merged subwords; plain chips are still single characters. Drag the slider or press play to watch the vocabulary build up — each step fuses the next most frequent pair, exactly how a tokenizer is trained.
The intuitions people miss
Eight things that surprise people
A token is not a word
One word can be several tokens. “tokenization” → “token” + “ization” — two tokens, one word.
Leading spaces matter
“world” and “ world” (with a space) are different tokens with different IDs. The space belongs to the word.
Numbers split unpredictably
“1000000” might become “100” + “0000”. The model rarely sees a clean number — one reason plain arithmetic is shaky.
Other languages cost more
The same sentence can be 8 tokens in English but 20+ in another script. Same meaning, higher token bill.
Emoji & symbols explode
Rare characters fall back to raw bytes, so a single emoji often costs several tokens.
Capitalization changes tokens
“Hello”, “hello”, and “HELLO” can be different tokens with different IDs. Shouting is literally a different sequence.
Code has its own logic
“->”, “=>”, “//” and indentation each become their own tokens. Code length in tokens rarely matches your intuition.
Whitespace is not free
Six spaces or “!!!!!!” become their own tokens instead of collapsing. Blank lines quietly burn budget.
How tokenizers are built
Start from bytes, repeatedly merge the most frequent adjacent pair until the vocabulary is full.
Like BPE, but merges the pair that best improves training likelihood; splits unknown words with a ## continuation mark.
Works on the raw stream with no splitting on spaces (marks spaces with ▁), so it is language-agnostic.
Start from a large candidate vocabulary and prune the tokens that least hurt likelihood, keeping a probabilistic best set.
Mental model: BPE and WordPiece build up by merging small pieces; Unigram trims down from a big set; SentencePiece is the space-handling framework the others can run inside.
Why it matters in production
- ▸Cost. Billing is per token — input plus output. Wordier prompts cost more, directly.
- ▸Context limits. Models have a hard token ceiling. Overflow and the earliest content becomes invisible.
- ▸Latency. More tokens means more compute, so longer answers take longer to stream.
- ▸Fairness. Languages that tokenize into more tokens cost their users more for the same content.
- ▸Prompt design. Knowing how text tokenizes lets you write tighter prompts and predict what will fit.
Rules of thumb
Related reading · LLM Fundamentals