Overview
Multi-Query RAG decomposes a complex question into several focused sub-queries, runs them in parallel against the vector index, then merges and deduplicates the results before generation. This recovers evidence on every dimension of a compound question in a single retrieval pass.
Pattern #25 (Parallel RAG) in the 37 RAG Patterns Collection
flowchart TD Q[Original query] --> MQ["LLM: generate N alternative phrasings of same intent max 4"] MQ --> Q1[Phrasing 1] & Q2[Phrasing 2] & Q3[Phrasing N] Q1 --> R1[Retrieve top-K] Q2 --> R2[Retrieve top-K] Q3 --> R3[Retrieve top-K] R1 & R2 & R3 --> MERGE[Merge all results] MERGE --> DEDUP[Deduplicate by chunk ID] DEDUP --> LLM[Generate]
The Problem
A compound question maps to multiple distinct information needs, but a single query only retrieves what is closest to its centroid:
Query: "Compare BERT and GPT on classification and generation tasks"
Single retrieval centroid ≈ "language model comparison"
→ Returns general comparison articles but misses:
- BERT classification benchmarks ❌
- GPT generation benchmarks ❌
- BERT architecture specifics ❌
- GPT fine-tuning details ❌
One embedding can't be simultaneously close to four distinct facets. The result is shallow coverage of each dimension and a generated answer that glosses over specifics.
Solution
Decompose first, then retrieve in parallel:
Original Query
↓
LLM Decomposition
↓
┌────────────┬────────────┬────────────┬────────────┐
Sub-query 1 Sub-query 2 Sub-query 3 Sub-query 4
"BERT "GPT "BERT "GPT
classification generation architecture fine-tuning"
accuracy" quality"
↓ ↓ ↓ ↓
Retrieve Retrieve Retrieve Retrieve (parallel)
└────────────┴────────────┴────────────┘
↓
Merge + Deduplicate
↓
Rerank merged set
↓
Generate Answer
Implementation
1. Query Decomposition
python
DECOMPOSE_PROMPT = """Break the following complex question into {n} focused sub-queries.
Each sub-query should target one specific aspect of the original question.
Sub-queries must be self-contained — each should retrieve useful results on its own.
Original question: {query}
Return a JSON list of strings: ["sub-query 1", "sub-query 2", ...]"""
def decompose_query(query, llm, n=4):
"""Decompose a complex query into focused sub-queries"""
prompt = DECOMPOSE_PROMPT.format(query=query, n=n)
result = llm.invoke(prompt, schema=SUBQUERY_SCHEMA)
return result["sub_queries"]2. Parallel Retrieval with asyncio
python
import asyncio
async def retrieve_async(sub_query, retriever, top_k=5):
"""Async retrieval for one sub-query"""
loop = asyncio.get_event_loop()
results = await loop.run_in_executor(
None, retriever.search, sub_query, top_k
)
return sub_query, results
async def parallel_retrieve(sub_queries, retriever, top_k=5):
"""Retrieve for all sub-queries concurrently"""
tasks = [
retrieve_async(q, retriever, top_k)
for q in sub_queries
]
return await asyncio.gather(*tasks)3. Deduplication and Merge
python
from sklearn.metrics.pairwise import cosine_similarity
import numpy as np
DEDUP_THRESHOLD = 0.95
def deduplicate_results(all_results, embedder):
"""Remove near-duplicate chunks using cosine similarity"""
seen_embeddings = []
unique_results = []
for sub_query, results in all_results:
for doc in results:
embedding = embedder.embed(doc["content"])
# Check similarity against already-kept chunks
if seen_embeddings:
sims = cosine_similarity(
[embedding], seen_embeddings
)[0]
if sims.max() >= DEDUP_THRESHOLD:
continue # Near-duplicate — skip
seen_embeddings.append(embedding)
unique_results.append({**doc, "source_query": sub_query})
return unique_results4. Full Pipeline
python
def multi_query_rag(query, retriever, embedder, reranker, llm):
# Step 1: Decompose into sub-queries
sub_queries = decompose_query(query, llm, n=4)
# Step 2: Retrieve in parallel (no sequential latency)
all_results = asyncio.run(
parallel_retrieve(sub_queries, retriever, top_k=5)
)
# Step 3: Merge and deduplicate
unique_docs = deduplicate_results(all_results, embedder)
# Step 4: Rerank merged set against original query
reranked = reranker.rerank(query, unique_docs, top_k=6)
# Step 5: Generate
context = "\n\n".join([d["content"] for d in reranked])
response = llm.generate(
f"Context: {context}\n\nQuestion: {query}"
)
return response, {
"sub_queries": sub_queries,
"raw_results": sum(len(r) for _, r in all_results),
"after_dedup": len(unique_docs),
"final_context": len(reranked),
}Complete Notebook
Run the full implementation:
Includes:
- ▸LLM-based query decomposition prompts
- ▸asyncio parallel retrieval implementation
- ▸Cosine-similarity deduplication at configurable threshold
- ▸Benchmark comparisons on single vs. multi-query
Performance
| Query Type | Multi-Query RAG | Single-Query RAG |
|---|---|---|
| Complex/compound questions | 76% | 51% |
| Comparison questions | 82% | 55% |
| Multi-facet research | 79% | 53% |
| Simple factoid questions | 84% | 83% |
Multi-query retrieval lifts accuracy on compound questions by 25 percentage points with no penalty on simple queries.
When to Use
Best For
- ▸Comparison questions — "How does A differ from B across dimensions X and Y?"
- ▸Multi-facet research — questions that span several distinct subtopics
- ▸"Explain A and B in the context of C" style queries
- ▸Long-form generation — answers that need comprehensive, multi-angle evidence
Combine With
- ▸RAG Fusion (#3) — apply Reciprocal Rank Fusion (RRF) to merge the ranked lists from each sub-query before deduplication
- ▸Cross-Encoder Reranking (#4) — rerank the merged, deduplicated set against the original query for final context selection
- ▸Contextual Compression (#6) — compress each retrieved chunk before merging to manage token budget
Related Papers
- ▸Multi-Query Retriever: LangChain Multi-Query Retriever (2023)
- ▸Step-Back Prompting: Take a Step Back (2023)
Next Steps
- ▸Try it: Run the Parallel RAG notebook
- ▸Tune: Experiment with N sub-queries — 3 is fast, 5 gives broader coverage
- ▸Combine: Add RRF from RAG Fusion (#3) for better rank merging
- ▸Compress: Apply Contextual Compression (#6) to keep token usage in check
Part of: 37 RAG Patterns Collection See also: Complete Pattern Index