Vector Indexes: How Search Engines Store and Retrieve Vectors

A vector index is not just an array of vectors with a search function. It handles persistence, concurrent reads and writes, index optimization, payload filtering, snapshots, and the HNSW graph structure that makes search fast. Understanding its internals helps you configure it correctly and debug it when things go wrong.

Core concepts

Collection — a named index for a set of vectors that share the same dimension and distance metric. Analogous to a table in a relational database.

Point — a single record: a unique ID, a vector, and an optional payload.

Payload — structured metadata stored alongside the vector (title, domain, tags, date, etc.). Used for filtering without embedding metadata into vectors.

Segment — the internal storage unit. A collection is divided into multiple segments. Segments are rebuilt during background optimization, which compacts deletions and improves HNSW graph quality.

Creating a collection

When creating a collection, you set three things that can't be changed later without re-indexing:

▸Dimension: must match your embedding model's output size exactly
▸Distance metric: cosine for text (almost always the right choice)
▸HNSW parameters: M and ef_construct trade off quality vs. build time

Sensible production defaults:

dimension    = 768         (match your embedding model)
distance     = cosine
M            = 16          (higher = better recall + more memory)
ef_construct = 256         (higher = better index quality + slower build)

Payload indexes for fast filtering

Store all metadata in the payload, not in the vector. Embedding metadata into vectors (e.g., prepending category names to text before encoding) pollutes the semantic space and makes re-indexing expensive.

Create explicit payload indexes on fields you'll filter by:

Field: "domain"   → keyword index   (exact match)
Field: "date"     → integer/date    (range queries)
Field: "tags"     → keyword index   (contains queries)

Without a payload index, filtering requires scanning all payloads — O(n). With an index, filtering is O(log n) on the indexed field.

Bulk ingestion strategy

Upload in batches of 100–200 points. Smaller batches increase API overhead; larger batches risk timeout on slow networks.

After bulk upload, wait for the index to finish optimizing before serving queries. Optimization merges small segments into larger ones and improves HNSW graph connectivity. Querying during optimization works but may return slightly degraded recall.

Pre-filtering vs. post-filtering

Pre-filtering applies the filter before HNSW traversal:

Filter → reduces search space → kNN on filtered subset

More efficient when the filter is selective (removes >30% of documents). The HNSW traversal only touches documents that pass the filter.

Post-filtering runs kNN first, then filters:

kNN (over-fetch 3–5×) → filter results → return top-k

Simpler but wasteful. Use only for very loose filters that rarely remove results.

[Key Insight] The pre-filtering / post-filtering decision is the most impactful configuration choice after HNSW parameters. A misconfigured filter strategy can double your latency. Benchmark both on your actual query patterns — don't rely on intuition.

Snapshot and backup

Take index snapshots before:

▸Any bulk re-index operation
▸Corpus migrations
▸Embedding model upgrades (which require full re-indexing)

A snapshot is a point-in-time copy of a collection. Restoration is fast — significantly faster than re-building the index from scratch.

Schedule automated snapshots in production. The cost is minimal (a few minutes for 1M docs); the recovery value if something goes wrong during optimization is high.

Handling deletes

Vector indexes with HNSW graphs don't delete cleanly. A deleted point leaves a "tombstone" in the graph — the node is marked deleted but its graph edges remain, which can slightly degrade recall over time.

For workloads with frequent deletes:

▸Use soft deletes (mark a payload field as "deleted")
▸Filter out soft-deleted results at query time
▸Periodically trigger a full re-index to compact tombstones

For append-only or rarely-updated corpora, this isn't a concern.

Index sizing

Memory requirements scale with corpus size, dimension, and HNSW M:

memory per doc ≈ (dims × 4 bytes) + (M × 2 × 4 bytes)
               = (768 × 4) + (16 × 2 × 4)
               = ~3.2 KB per document

1M documents   ≈ 3.2 GB
10M documents  ≈ 32 GB

Use INT8 quantization to cut vector storage 4×. Keep the quantized vectors in RAM; store original FP32 vectors on disk for rescoring. This reduces RAM requirements while preserving recall through the two-phase retrieval pattern.

Choosing between dedicated vector index vs. SQL extension

Dedicated vector index (purpose-built for vectors):

▸Best for corpora >500K documents
▸Higher throughput and better HNSW tuning control
▸Separate operational surface to manage

SQL extension (vector search inside a relational database):

▸Best for corpora <500K documents
▸Single operational surface if you're already running a relational database
▸Supports rich joins between vectors and relational data
▸HNSW tuning options are more limited

If you're starting fresh and expect to exceed 1M documents, a dedicated vector index is the right foundation. If you're integrating vector search into an existing relational system and your corpus is bounded, the SQL extension route is simpler to operate.