SPEC: Embeddings
Definition
[Definition] Embeddings are dense numerical vectors that represent tokens (or sentences, documents, images) in a continuous high-dimensional space. They encode semantic meaning so that similar concepts are geometrically close to each other.
Why Embeddings?
- ▸Computers can't process raw text — they need numbers
- ▸One-hot encoding (10,000-dim sparse vector) is inefficient and captures no meaning
- ▸Embeddings are compact (256–4096 dims) and encode rich semantic relationships
How Token Embeddings Work
- ▸Each token ID maps to a row in an embedding matrix (shape:
vocab_size × embed_dim) - ▸At inference, the model does a simple lookup: token_id → embedding vector
- ▸This embedding matrix is learned during pre-training
- ▸The same matrix is often used (transposed) at the output layer to predict the next token (weight tying)
Dimensions (Typical Values by Model Size)
| Model Size | Embedding Dimension |
|---|---|
| Small (125M) | 768 |
| Medium (1.3B) | 2048 |
| Large (7B) | 4096 |
| XL (70B+) | 8192 |
Properties of Good Embeddings
- ▸Semantic similarity: "king" and "queen" are close; "king" and "car" are far
- ▸Arithmetic:
king - man + woman ≈ queen(Word2Vec famous example) - ▸Contextual vs. static:
- ▸Static (Word2Vec, GloVe): one fixed vector per word regardless of context
- ▸Contextual (BERT, GPT): each occurrence gets a different vector based on surrounding tokens
Types of Embeddings
| Type | Description | Use Case |
|---|---|---|
| Token embeddings | Per-token lookup vectors | Input to every transformer |
| Positional embeddings | Encode position in sequence | Combined with token embeddings |
| Sentence embeddings | Single vector for entire sentence | Semantic search, RAG |
| Image embeddings | Encode visual content | Multimodal models |
| Document embeddings | Encode entire documents | Long-doc retrieval |
Positional Embeddings
Transformers have no inherent notion of order — all tokens are processed in parallel. Positional embeddings add position information:
- ▸Absolute (sinusoidal) — original Transformer, fixed sine/cosine patterns
- ▸Learned absolute — trained position vectors (BERT, GPT-2)
- ▸Relative (RoPE) — Rotary Position Embedding, encodes relative distance; used by LLaMA, Mistral, GPT-NeoX
- ▸ALiBi — adds a linear bias to attention based on distance; good for length generalization
Embedding Space (Latent Space)
[Key Insight] - The full embedding space is also called latent space - High-dimensional geometry encodes language structure - Nearest-neighbor search in this space = semantic search - Dimensionality reduction (t-SNE, UMAP) used to visualize clusters
Practical Uses of Embeddings
- ▸Semantic search: embed query + documents → cosine similarity → retrieve relevant docs
- ▸RAG (Retrieval-Augmented Generation): store doc embeddings in vector DB, retrieve at query time
- ▸Clustering: group similar documents
- ▸Classification: feed embedding into a classifier head
- ▸Anomaly detection: find outliers in embedding space
Similarity Metrics
| Metric | Formula | Notes |
|---|---|---|
| Cosine similarity | `cos(θ) = A·B / ( | A |
| Dot product | A·B | Fast, used in attention |
| Euclidean distance | √Σ(ai-bi)² | Magnitude-sensitive |
Popular Embedding Models
- ▸
text-embedding-3-large(OpenAI) — 3072 dims - ▸
amazon.titan-embed-text-v2(AWS Bedrock) - ▸
all-MiniLM-L6-v2(HuggingFace/SentenceTransformers) — lightweight - ▸
nomic-embed-text— open source, long context
Related Concepts
- ▸Token, Tokenization, Latent Space, Attention, RAG, Vector Database