Embeddings: Turning Text into Vectors
An embedding is a fixed-length array of numbers that encodes the meaning of a piece of text. The model that produces embeddings is trained so that semantically similar texts produce numerically similar vectors — and dissimilar texts produce distant ones.
How embedding models work
Embedding models are transformer networks with one key modification: instead of predicting the next token, they output a single pooled vector representing the entire input sequence.
Training uses contrastive learning — the model is shown millions of sentence pairs labeled as "similar" or "different". It learns to pull similar pairs close and push different pairs apart. After training on hundreds of millions of such pairs, the model has learned a geometry where proximity = shared meaning.
Input: "The battery lasts all day"
Output: [0.23, -0.87, 0.41, 0.12, ...] ← 768 numbers
Input: "Long battery life" ← semantically identical
Output: [0.22, -0.84, 0.40, 0.11, ...] ← nearly identical vector
Input: "GPU memory bandwidth" ← completely unrelated
Output: [0.94, 0.12, -0.73, 0.55, ...] ← far away in vector space
Context changes the output
The same word produces different vectors depending on context — this is what separates modern embeddings from older bag-of-words approaches:
"Apple announced record profits" → vector A (company context)
"I ate a fresh apple this morning" → vector B (fruit context)
Vectors A and B are not similar. The model encodes the full sentence, not individual tokens in isolation.
Dimensions: quality vs. cost
Higher dimensions = more representational capacity, but more memory and slower distance calculations:
| Dimensions | Memory per 1M docs | Relative quality | Best for |
|---|---|---|---|
| 384 | ~1.5 GB | Good | High-throughput, cost-sensitive |
| 768 | ~3 GB | Better | Production default |
| 1024 | ~4 GB | Best | Highest-recall use cases |
Diminishing returns kick in past 768 for most text search tasks. Start at 768 unless you have a specific reason to go higher.
Normalizing embeddings
Before storing or comparing vectors, normalize them to unit length (L2 norm = 1):
normalized_vec = vec / ||vec||
This has two benefits:
- ▸Cosine similarity reduces to a plain dot product, which is faster to compute
- ▸Results are consistent regardless of document length
Always normalize at both index time and query time using the same procedure.
Chunking long documents
Most embedding models have a maximum input length (typically 512 tokens). For longer documents, split into overlapping chunks before embedding:
Document: [word_1, word_2, ..., word_1200]
Chunk 1: words 1–400
Chunk 2: words 350–750 ← 50-word overlap preserves context at boundaries
Chunk 3: words 700–1100
Chunk 4: words 1050–1200
Store the chunk-to-document mapping so you can retrieve the original document after finding a matching chunk. Never silently truncate — you'll lose the end of every long document.
Caching embeddings
Embedding generation is the expensive step — the rest of the pipeline is fast. Cache everything:
- ▸Compute embeddings once per document, store alongside the document
- ▸Never re-embed unchanged content
- ▸If your corpus updates incrementally, only embed new or changed documents
A cache hit is zero cost. A cache miss costs one model inference.
Batch processing for throughput
Single-document encoding is fine at query time. For indexing a corpus, batch for throughput:
batch_size = 32 ← typical sweet spot for CPU
64–128 ← better on GPU with enough VRAM
Process corpus in batches of 32–64 → 3–5x faster than one-by-one
Show a progress bar during corpus indexing — it helps catch silent failures mid-batch (OOM, malformed inputs).
[Key Insight] The single most common mistake with embeddings in production: mismatching models between index time and query time. If you embedded your corpus with model A, you must embed queries with model A. Different models produce different vector spaces — cross-model similarity is meaningless.
What preprocessing to apply
Before embedding, clean text consistently:
- ▸Strip leading/trailing whitespace
- ▸Collapse multiple spaces into one
- ▸Remove non-content markup (HTML tags, formatting artifacts)
- ▸Apply the same cleaning at both index time and query time
Inconsistent preprocessing is the second most common cause of poor search results, after model mismatches.