Query Routing - Send Each Query to the Right Retriever

Overview

Query Routing classifies each incoming query and dispatches it to the most appropriate retrieval backend — vector store, SQL database, knowledge graph, or live web search. Instead of forcing every question through the same index, the router matches query intent to the backend best equipped to answer it.

Pattern #8 (Adaptive RAG) in the 37 RAG Patterns Collection

flowchart TD
  Q[Incoming query] --> CL["LLM classifier: detect query type max_tokens=10"]
  CL -- factual precise --> VS[Dense vector search]
  CL -- keyword / code --> KS[Keyword BM25 search]
  CL -- comparative / broad --> HY[Hybrid search BM25 + vector]
  CL -- structured / filtered --> META[Metadata-filtered search]
  VS & KS & HY & META --> LLM[Generate]

The Problem

One retrieval strategy doesn't fit all query types:

A semantic vector index excels at conceptual similarity but cannot compute aggregates, traverse entity graphs, or fetch content published after indexing. Routing every query to the same backend leaves structured, relational, and real-time questions systematically under-served.

Solution

Classify the query first, then send it to the right backend:

Route types:

• ▸vector — conceptual, semantic, or explanatory questions ("explain transformer attention")
• ▸sql — numeric, aggregation, or filter questions ("sales in Q3 by region")
• ▸graph — relational or entity-relationship questions ("who reports to the CTO?")
• ▸web — real-time or post-index questions ("latest model releases this week")
• ▸hybrid — ambiguous or compound queries that benefit from multiple backends

Implementation

1. Query Classifier

ROUTING_PROMPT = """You are a query router. Classify the query into exactly one route.

Routes:
- vector: conceptual, semantic, or explanatory questions
- sql: numeric, statistical, or structured data questions
- graph: relational, organizational, or entity-relationship questions
- web: real-time, news, or recent-event questions
- hybrid: complex queries requiring multiple backends

Query: {query}

Return JSON: {{"route": "<route>", "reasoning": "<brief explanation>", "confidence": <0.0-1.0>}}"""

def classify_query(query, llm):
    """Classify query intent and select retrieval route"""
    prompt = ROUTING_PROMPT.format(query=query)
    result = llm.invoke(prompt, schema=ROUTE_SCHEMA)
    return result

2. Backend Dispatcher

def route_to_backend(route, query, backends):
    """Dispatch query to the appropriate retrieval backend"""
    if route == "vector":
        return backends["vector"].search(query, top_k=5)

    elif route == "sql":
        sql = backends["text_to_sql"].generate(query)
        return backends["sql"].execute(sql)

    elif route == "graph":
        cypher = backends["text_to_cypher"].generate(query)
        return backends["graph"].run(cypher)

    elif route == "web":
        return backends["web_search"].search(query, num_results=5)

    elif route == "hybrid":
        vector_results = backends["vector"].search(query, top_k=3)
        web_results = backends["web_search"].search(query, num_results=3)
        return vector_results + web_results

    else:
        raise ValueError(f"Unknown route: {route}")

3. Full Pipeline with Confidence Fallback

CONFIDENCE_THRESHOLD = 0.7

def query_routing_rag(query, backends, llm):
    # Step 1: Classify query intent
    classification = classify_query(query, llm)
    route = classification["route"]
    confidence = classification["confidence"]

    # Step 2: Low-confidence fallback — try multiple routes and merge
    if confidence < CONFIDENCE_THRESHOLD:
        vector_results = backends["vector"].search(query, top_k=3)
        web_results = backends["web_search"].search(query, num_results=2)
        results = vector_results + web_results
        route = "hybrid-fallback"
    else:
        # Step 3: Route to the correct backend
        results = route_to_backend(route, query, backends)

    # Step 4: Generate answer from retrieved context
    context = format_results(results, route)
    response = llm.generate(
        f"Context: {context}\n\nQuestion: {query}"
    )

    return response, {
        "route": route,
        "confidence": confidence,
        "reasoning": classification["reasoning"],
        "num_results": len(results),
    }

Complete Notebook

Run the full implementation:

Adaptive RAG AWS Notebook

Includes:

• ▸LLM-based query classifier with routing prompt
• ▸Vector, SQL, graph, and web backend connectors
• ▸Confidence-threshold fallback with result merging
• ▸Benchmark comparisons across query types

Performance

Routing lifts mixed-benchmark accuracy by 29 percentage points without sacrificing semantic query quality.

When to Use

Best For

• ▸Multi-source knowledge bases — systems with both document stores and databases
• ▸Enterprise search — combining SQL analytics, graph org charts, and semantic docs
• ▸Mixed query workloads — users ask both "what is X" and "how many X last quarter"
• ▸Real-time knowledge gaps — some queries need content that postdates the index

Combine With

• ▸Ensemble Retrieval (#19) — use as fallback when routing confidence is low; merge results from multiple backends
• ▸Adaptive RAG (#8) — extend routing to full strategy selection (chunk size, retrieval depth, reranking)
• ▸Reranking (#4) — apply cross-encoder reranking after hybrid-fallback merges multiple backend results

Related Papers

• ▸Self-Route: Self-Routing RAG (2024)
• ▸RouteLLM: RouteLLM: Learning to Route LLMs with Preference Data (2024)

Next Steps

1. ▸Try it: Run the Adaptive RAG notebook
2. ▸Expand: Add a graph backend for entity-relationship queries
3. ▸Monitor: Log route decisions and confidence scores to detect misclassifications
4. ▸Combine: Wire uncertain routes to Ensemble Retrieval (#19) as a safety net

Part of: 37 RAG Patterns Collection See also: Complete Pattern Index