SPEC: Latent Space
Definition
[Definition] Latent space is the high-dimensional mathematical space in which embeddings are organized. It is a learned, continuous representation space where the geometric positions of vectors encode meaning — similar concepts cluster together, and relationships between concepts are reflected by distances and directions.
The Word "Latent"
"Latent" means hidden or underlying. The latent space is not directly observed — it is the model's internal compressed representation of the world, learned entirely from data patterns.
Structure of Latent Space
- ▸Every token, word, sentence, or concept maps to a point (vector) in this space
- ▸The space has hundreds to thousands of dimensions (e.g., 4096 for LLaMA-3 7B)
- ▸Directions in this space are meaningful:
- ▸There may be a "gender" direction:
king - man + woman ≈ queen - ▸A "capital city" direction:
France - Paris + Berlin ≈ Germany - ▸A "sentiment" direction: positive sentiment tokens cluster in one region
- ▸There may be a "gender" direction:
How It's Built
- ▸During pre-training, the model adjusts its embedding matrix and attention weights to minimize next-token prediction loss
- ▸Tokens that appear in similar contexts get pulled toward each other in the space
- ▸Over billions of training steps, the space organizes itself to reflect the statistical structure of language
Geometric Intuitions
| Geometric Property | Semantic Meaning |
|---|---|
| Small distance (cosine similarity ≈ 1) | Semantically similar |
| Large distance (cosine similarity ≈ 0) | Semantically unrelated |
| Vector arithmetic | Analogical relationships |
| Clusters | Conceptual categories (animals, countries, verbs...) |
| Manifolds | Underlying structure of a concept domain |
Layers of Latent Space in a Transformer
A Transformer has multiple layers, and each produces its own latent representations (called hidden states):
- ▸Early layers: syntactic structure (POS, morphology)
- ▸Middle layers: semantic structure (entity types, coreference)
- ▸Late layers: task-specific, output-oriented representations
- ▸The final layer hidden states feed into the output (LM) head
Latent Space vs. Embedding Space
These terms are often used interchangeably but have a subtle distinction:
| Term | Meaning |
|---|---|
| Embedding Space | The input embedding lookup table (static-ish) |
| Latent Space | The full internal representational space after all transformer layers |
| Hidden State | A vector in the latent space at a specific layer |
Applications
Semantic Search
- ▸Encode query + documents into latent space
- ▸Use cosine similarity to find nearest neighbors
- ▸Powers RAG, recommendation engines, deduplication
Visualization
- ▸t-SNE and UMAP project high-dim latent space → 2D/3D
- ▸Reveals clusters, outliers, and structure in data
Probing
- ▸Train small classifiers on hidden states to discover what information each layer encodes
- ▸Example: Does layer 12 know whether a word is a proper noun? Probe it.
Interpolation
- ▸Interpolating between two points in latent space can generate smooth transitions between concepts (common in image generation, less so in text)
Steering Vectors
- ▸Identify directions in latent space corresponding to behaviors (e.g., "sycophancy", "refusal")
- ▸Activate or suppress behavior by adding/subtracting these vectors at inference time (activation steering / representation engineering)
Why Latent Space Matters for LLM Practitioners
[Key Insight] - RAG quality: how well your retrieval works depends entirely on the quality of the embedding model's latent space - Fine-tuning: fine-tuning shifts the latent space to accommodate new task structure - Interpretability: understanding the latent space is central to mechanistic interpretability research - Vector DBs: storing and querying vectors is storing and querying latent space points
Related Concepts
- ▸Embeddings, Attention, Hidden States, RAG, Semantic Search, Fine-Tuning, Activation Steering