Positional Encoding: How Transformers Know Word Order
[Definition] Positional encoding supplies sequence order to a Transformer. Self-attention compares token representations in parallel; without a position signal, it cannot distinguish “dog bites man” from “man bites dog.”
The problem: attention sees a set, not a sequence
A Transformer starts with token embeddings, then applies the same attention operation to every row. If the rows are permuted, attention has no built-in clock, index, or left-to-right direction.
| Sequence | Meaning |
|---|---|
dog bites man | the dog is the actor |
man bites dog | the man is the actor |
not approved | rejection |
approved, not | a different and malformed meaning |
The input must combine what token this is with where it is:
xₚ = token_embedding(tokenₚ) + position_signal(p)Sinusoidal positional encoding
The original Attention Is All You Need paper used fixed sine and cosine waves. For position p and embedding dimension i:
PE(p, 2i) = sin(p / 10000^(2i / d_model))
PE(p, 2i + 1) = cos(p / 10000^(2i / d_model))Each pair of dimensions oscillates at a different frequency:
- ▸high-frequency dimensions distinguish nearby tokens;
- ▸low-frequency dimensions carry coarse location;
- ▸together they create a smooth signature for each position.
Position: 0 1 2 3 4
Fast wave: 0 0.84 0.91 0.14 -0.76
Slow wave: 0 0.01 0.02 0.03 0.04The encoding is deterministic: there is no position table to train, and the formula can be evaluated for an index not seen during training.
Why waves help attention reason about distance
The useful property is not merely that every row is unique. Sine and cosine let a model express a shift using linear combinations:
sin(p + k) = sin(p)cos(k) + cos(p)sin(k)An attention head can learn relative patterns such as “look one token to the left,” “attend to the opening bracket,” or “find a subject roughly ten tokens before this verb.”
Absolute, learned, and relative positions
| Method | How it works | Strength | Limitation |
|---|---|---|---|
| Sinusoidal | Add fixed waves to token embeddings | no learned table; works at new indices | absolute location is mixed into every vector |
| Learned absolute | learn one vector per position | flexible inside training range | fixed maximum position table |
| Relative bias | add a score bias based on distance | makes distance explicit | needs a range/bias scheme |
| RoPE | rotate queries and keys by position | relative offset emerges in dot products | long-context scaling needs care |
| ALiBi | penalize attention by distance | simple and length-friendly | less expressive than full learned bias |
RoPE: rotary position embeddings
Most modern decoder-only LLMs use Rotary Position Embeddings (RoPE). Rather than adding a vector only at the input, RoPE rotates pairs of dimensions in queries and keys:
q'ₚ = R(p)qₚ
k'ₜ = R(t)kₜ
attention score = q'ₚ · k'ₜThe dot product naturally depends on the relative offset p - t. That is what attention usually needs: not “this token is at 4,096,” but “this key is twelve tokens before my query.”
[Key Insight] RoPE encodes position where attention uses it—inside query/key geometry—so relative order is available directly to attention scores.
Long context is not automatically solved
A model trained at 4K tokens does not reliably become a 128K-token model just because a formula can produce larger positions. At longer distances, rotations occur in patterns the model did not learn and evidence can be lost in the middle of the context.
Long-context systems therefore use RoPE scaling or interpolation, continued long-context training, and evaluation at multiple answer positions. Context-window size is an engineering and evaluation problem, not a single configuration field.
Production implications
Chunking changes the position problem
In RAG, a chunk often enters the model with a fresh position zero. The model knows order inside the chunk but not its original document location. Include section title, page number, parent heading, and neighbouring context when location matters.
Caching does not remove positions
The KV cache stores keys and values that already contain positional information. A new generated token must use the correct position; an off-by-one cache index produces wrong attention rather than a clear failure.
Evaluate at the target context length
If a task needs 32K-token evidence packs, evaluate retrieval and answer quality near 32K. Performance at 2K tokens is not evidence that ordering behaviour will hold at the required production length.
Mental model
Token embedding: “bank” may mean a financial institution or river edge
Position signal: this “bank” follows “river” and precedes “was steep”
Attention: use both meaning and location to select contextToken identity tells the model what is available. Positional encoding tells the model how those pieces are arranged.
Continue the path
This public article is the conceptual anchor. The Pro Transformer Architecture sequence builds positional encoding with tensors and computed values, then connects it to attention, masking, and generation. See the learning roadmap for the full path.