Quantization: Cut Vector Memory by 4–40x

At 768 dimensions, each vector uses 3,072 bytes (768 × 4 bytes, FP32). One million documents = 3 GB. Ten million = 30 GB. At some point, the RAM cost of exact FP32 vectors becomes the bottleneck.

Quantization trades a small amount of recall for a large reduction in memory. The tradeoff is almost always worth it — especially INT8.

Method	Memory reduction	Recall@10 drop	Speed impact
FP32 (baseline)	1×	—	baseline
FP16	2×	< 0.1%	negligible
INT8	4×	~0.5–1%	~10% faster
Binary	32–40×	~5–8%	40× faster

FP16: the free win

Half-precision (16-bit) floats use 2 bytes instead of 4. For cosine similarity ranking, the precision difference is negligible — the relative ordering of nearest neighbors barely changes.

FP16 is the minimum quantization you should apply. There is almost no downside.

INT8: 4× smaller with minimal recall loss

INT8 maps each float value to a signed 8-bit integer (−128 to 127). The mapping is derived from your corpus's value distribution:

quantized_value = round((original - min) / (max - min) × 255) - 128

Clipping extreme outliers (top and bottom 0.5–1%) before mapping improves the mapping for typical values — this is the quantile parameter.

In practice, INT8 reduces recall@10 by about 0.5–1% while cutting memory 4×. For a 10M document corpus, this means 30 GB → 7.5 GB. That's the difference between a $200/month and a $800/month server.

INT8 is the right default for production. Apply it at index creation — retrofitting requires re-indexing the entire corpus.

Binary quantization: 40× smaller

Binary quantization maps each dimension to a single bit: positive → 1, negative → 0. The 768-dimension vector becomes 96 bytes (768 ÷ 8) instead of 3,072 bytes.

Distance is computed with XOR + popcount — extremely fast on modern CPUs. This is 40× faster than FP32 dot products.

The recall drop (5–8%) sounds large, but the rescore pattern recovers it:

The rescore pattern

Two-phase retrieval is the key to making binary quantization production-viable:

Phase 1: Binary quantized search
  - Retrieve top-100 candidates (very fast, cheap)
  - Recall@100 is still high even with quantization

Phase 2: FP32 rescore
  - Compute exact cosine similarity for the 100 candidates
  - Return top-10 with accurate rankings

The final recall@10 with rescoring is within 1–2% of pure FP32, while the memory savings are 32–40×. The two-phase approach only adds the cost of 100 FP32 dot products — microseconds.

[Definition] Keep FP32 vectors on disk, quantized vectors in RAM. The index uses quantized vectors for fast candidate retrieval; the rescore step reads the original FP32 vectors for the small candidate set. This gives you memory efficiency and accuracy.

When quantization hurts

Avoid aggressive quantization when:

▸Corpus < 50K documents: memory isn't the bottleneck, FP32 overhead is minimal
▸Recall > 99% required: medical, legal, financial applications where missing a result has real consequences
▸Fine-grained distinction critical: embedding models trained on highly specialized domains may rely on subtle float differences that quantization destroys

For most knowledge base, documentation, e-commerce, and support search applications, INT8 is correct. Enable it at collection creation.

Dimensionality reduction as an alternative

Instead of quantizing precision, reduce the number of dimensions:

768-dimension vector → PCA → 256-dimension vector (3× smaller)

PCA preserves roughly 90–95% of variance in 256 dimensions. The recall loss is similar to INT8, but it applies at both index time and query time — you must reduce query vectors with the same PCA model.

Use PCA for:

▸Visualization (project to 2D/3D for t-SNE plots)
▸Cross-modal search where lower-dim alignment is beneficial

Don't use PCA as a substitute for INT8 in pure text search — INT8 is simpler to operate and has better-characterized recall behavior.

Summary: what to apply

For any production deployment:

▸Always: normalize vectors to unit length before indexing
▸Default: INT8 scalar quantization (4× memory, ~1% recall drop)
▸Large corpora (>5M docs): binary quantization + FP32 rescore (32× memory, ~1% final recall drop)
▸FP16: apply as a first step regardless of other quantization — no meaningful tradeoff