The Evolution of Search
Search went from matching letters to understanding meaning to reasoning. Walk the four eras, then see the two ways to encode meaning and what vectors are really for.
🧭 One question, four answers
Every search era exists because the previous one hit a wall. Keyword search is fast but blind to meaning. Semantic search understands meaning but stumbles on exact codes. Hybrid combines both. Agentic search lets an AI plan the whole thing. Click through the timeline to see what each era solves — and where it still fails.
Hybrid
Best of BothCombines keyword and semantic in parallel with score normalization.
Strengths
- ✓BM25 + semantic in parallel
- ✓Score normalization / fusion
- ✓State-of-the-art today
Limitations
- •Two pipelines to tune
- •Needs fusion weighting (e.g. RRF)
- •More moving parts
Semantic search — two approaches
Once you go semantic, there are two ways to turn meaning into something searchable.
Sparse Encoding
- • Expands documents with weighted synonyms (“car” → automobile 0.8, vehicle 0.6)
- • Uses the existing inverted index (Lucene-native)
- • 30,522 dims (BERT vocab) — only ~1% non-zero per doc
- • 105,879 dims for the multilingual variant
- • As efficient as BM25 at query time
- • Pre-trained: opensearch-neural-sparse-v2
Dense Encoding
- • Fixed-length continuous vectors (384–768 dim)
- • Stored in a vector index (HNSW, IVF, etc.)
- • Captures deep semantic relationships in vector space
- • k-NN search at query time
- • BERT, Cohere, OpenAI, Amazon Titan, Bedrock
What do I do with vectors?
Any content — text, images, audio — becomes a vector via an embedding model. Then a vector engine lets you:
Similarity retrieval
Find content most similar to a question, image, or music clip.
Classification
Content classification, salient terms, and topic extraction.
Keyword relevance
Retrieve most relevant content by key terms and metadata.
image · documents · audio → embedding model → [0.35, 0.1, 0, 0.9, …] → vector engine
Dense encoding — the full power of embeddings
Vector Space
Text becomes fixed-length vectors. Similarity via cosine, dot product, or L2. Captures deep semantics and nuance.
AI-Ready
Powers RAG pipelines with LLMs. Cross-lingual search capability. Works with BERT, Cohere, and Titan.
Optimized
HNSW, IVF, IVFPQ engines. FP16, INT8, and binary quantization — 50–97% memory reduction.
See it in action
The Semantic Search lab runs the “semantic” era live with real Amazon Titan embeddings. The RAG Readiness Scan checks whether your documents are even ready for this pipeline.
Related reading · Knowledge Lab