SPEC: Tokenization

Definition

[Definition] Tokenization is the process of converting raw text into a sequence of tokens — the discrete numeric IDs that an LLM can process. It is the first and last step in every LLM pipeline: text → token IDs (encoding) and token IDs → text (decoding).

The Full Pipeline

Steps in Tokenization

1. ▸Normalization — Unicode normalization, lowercasing (model-dependent), whitespace handling
2. ▸Pre-tokenization — split on spaces/punctuation as initial boundary hints
3. ▸Subword splitting — apply BPE/WordPiece/SentencePiece rules to produce final token units
4. ▸Vocabulary lookup — map each token string → integer ID
5. ▸Special token injection — add [BOS], [EOS], [PAD] as required by the model

Major Algorithms

Byte Pair Encoding (BPE)

• ▸Start with individual characters as vocabulary
• ▸Repeatedly merge the most frequent adjacent pair
• ▸Stop when vocabulary reaches target size
• ▸Used by: GPT family, LLaMA, Mistral

WordPiece

• ▸Similar to BPE but merges based on maximizing likelihood of training data
• ▸Unknown tokens split with ## prefix for continuations
• ▸Used by: BERT, DistilBERT

SentencePiece

• ▸Treats input as a raw byte stream (language-agnostic)
• ▸Supports both BPE and unigram language model modes
• ▸Handles spaces explicitly with ▁ symbol
• ▸Used by: T5, XLNet, Gemini

Unigram Language Model

• ▸Starts with a large vocabulary and prunes tokens that minimize loss increase
• ▸More probabilistic approach

Encoding vs. Decoding

Why Tokenization Matters

[Key Insight] - Model performance: poor tokenization = poor understanding of rare words, numbers, code - Multilingual support: byte-level tokenizers handle all languages; word-level struggles with non-Latin scripts - Arithmetic tasks: models struggle with math partly because numbers tokenize inconsistently ("123" may be 1 or 3 tokens) - Prompt engineering: knowing how text tokenizes helps design efficient, precise prompts

Common Gotchas

• ▸Leading spaces: " word" ≠ "word" as tokens
• ▸Numbers: "1000000" may tokenize as ["100", "0000"] — not ["1000000"]
• ▸Code symbols: ->, =>, // each tokenize differently per tokenizer
• ▸Emoji/Unicode: may tokenize into many byte-level tokens

Tools

• ▸tiktoken (OpenAI) — fast Python tokenizer for GPT models
• ▸transformers.AutoTokenizer (HuggingFace) — universal tokenizer loader
• ▸Tokenizer Playground (OpenAI) — visual token inspection

Related Concepts

• ▸Token, Embeddings, Vocabulary, Context Window, BPE