Advanced10 min read21 of 40

Agentic RAG - Autonomous Decision Making

Empower LLMs to autonomously decide when to retrieve, what to search, and how to combine information. Advanced pattern for complex multi-step reasoning.

Read first:Simple RAGSelf-RAG

Overview

Agentic RAG gives the LLM autonomous control over the retrieval process. The model decides:

  • ▸When to retrieve (is current knowledge sufficient?)
  • ▸What to search for (formulate sub-queries)
  • ▸How to combine results (synthesis strategy)

Pattern #12 in the 37 RAG Patterns Collection

Architecture

graph TB
    A[User Query] --> B{Agent: Need Retrieval?}
    B -->|Yes| C[Agent: Plan Search Strategy]
    B -->|No| G[Direct Answer]
    
    C --> D[Execute Searches]
    D --> E{Agent: Sufficient?}
    E -->|No| C
    E -->|Yes| F[Agent: Synthesize]
    
    F --> G[Final Response]
    
    H[Tools: Search, Retrieve, Filter] --> D
    
    style A fill:#f9f,stroke:#333
    style B fill:#ff9,stroke:#333
    style E fill:#ff9,stroke:#333
    style G fill:#9f9,stroke:#333

Core Capabilities

1. Autonomous Retrieval Decision

python
def should_retrieve(query, llm_knowledge):
    """Agent decides if retrieval is needed"""
    prompt = f"""Query: {query}

Based on your knowledge, can you answer this accurately without additional information?

Respond with JSON:
{{
    "needs_retrieval": true/false,
    "reasoning": "explanation",
    "confidence": 0-100
}}"""
    
    decision = llm.invoke(prompt, schema=DECISION_SCHEMA)
    return decision['needs_retrieval']

2. Dynamic Query Formulation

python
def formulate_searches(query):
    """Agent creates optimal search strategy"""
    prompt = f"""Query: {query}

Break this into targeted search queries that would find the needed information.

Return JSON:
{{
    "searches": [
        {{"query": "...", "type": "vector/keyword/graph"}},
        ...
    ],
    "reasoning": "why these searches"
}}"""
    
    return llm.invoke(prompt, schema=SEARCH_SCHEMA)

3. Iterative Refinement

python
def iterative_retrieval(query, max_iterations=3):
    """Agent refines search based on results"""
    context = []
    
    for i in range(max_iterations):
        # Evaluate current context
        evaluation = llm.invoke(f"""
Context gathered: {context}
Original query: {query}

Is this sufficient to answer accurately?
If not, what additional information is needed?
""", schema=EVAL_SCHEMA)
        
        if evaluation['sufficient']:
            break
        
        # Get more information
        new_searches = formulate_searches(evaluation['gaps'])
        new_results = execute_searches(new_searches)
        context.extend(new_results)
    
    return context

Complete Implementation

python
class AgenticRAG:
    def __init__(self, llm, retriever, tools):
        self.llm = llm
        self.retriever = retriever
        self.tools = tools  # search, filter, rank, etc.
    
    def query(self, user_query):
        # Phase 1: Assess need
        if not self.should_retrieve(user_query):
            return self.llm.generate(user_query)
        
        # Phase 2: Plan strategy
        plan = self.create_search_plan(user_query)
        
        # Phase 3: Execute with autonomy
        context = []
        for step in plan['steps']:
            # Agent chooses tool and parameters
            tool = self.select_tool(step)
            results = tool.execute(step['parameters'])
            
            # Agent evaluates results
            if self.are_results_useful(results, user_query):
                context.extend(results)
            
            # Agent decides: continue or done?
            if self.has_sufficient_info(context, user_query):
                break
        
        # Phase 4: Synthesize
        return self.synthesize_answer(context, user_query)
    
    def select_tool(self, step):
        """Agent chooses best tool for the task"""
        prompt = f"""Task: {step['description']}

Available tools: {self.tools.keys()}

Which tool is best? Return tool name and parameters."""
        
        selection = self.llm.invoke(prompt, schema=TOOL_SCHEMA)
        return self.tools[selection['tool']]

Complete Notebook

🚀 Run the full implementation:

📓 Agentic RAG AWS Notebook

Includes:

  • ▸Complete agent framework
  • ▸Tool selection logic
  • ▸Iterative refinement
  • ▸Multi-step reasoning examples
  • ▸Performance benchmarks

Comparison

FeatureAgentic RAGStandard RAG
AutonomyFullNone
AdaptabilityDynamicFixed pipeline
Accuracy90%+70%
LatencyVariable (0.5-5s)Fixed (~300ms)
Cost$0.18-0.50$0.08
ComplexityHighLow

When to Use

Best For

  • ▸Complex research questions requiring multi-step reasoning
  • ▸Open-ended queries without clear retrieval strategy
  • ▸Dynamic domains where fixed pipelines fail
  • ▸High-value applications where accuracy justifies cost

Avoid When

  • ▸Simple Q&A (unnecessary complexity)
  • ▸Strict latency requirements (unpredictable timing)
  • ▸Budget constraints (multiple LLM calls)
  • ▸Production at scale (complex failure modes)

Agent Tools

Common tools provided to agentic RAG:

python
AGENT_TOOLS = {
    "vector_search": VectorSearchTool(),
    "keyword_search": KeywordSearchTool(),
    "graph_traverse": GraphTraversalTool(),
    "web_search": WebSearchTool(),
    "filter": FilterTool(),
    "rerank": RerankingTool(),
    "summarize": SummarizationTool(),
}

Example Flow

Query: "Compare the economic impact of renewable energy adoption in Germany vs China from 2015-2025"

Agent's Decision Process

  1. ▸Assess: "This requires specific data I don't have → need retrieval"
  2. ▸Plan:
    • ▸Search: "Germany renewable energy economic impact 2015-2025"
    • ▸Search: "China renewable energy economic impact 2015-2025"
    • ▸Search: "renewable energy economics comparison methodology"
  3. ▸Execute:
    • ▸Run searches → get initial results
    • ▸Evaluate: "Missing GDP impact data" → refine search
    • ▸Add search: "Germany GDP renewable energy contribution"
    • ▸Evaluate: "Now have sufficient data" → stop
  4. ▸Synthesize: Compare findings, structure answer

Optimization Tips

1. Limit Iterations

python
MAX_ITERATIONS = 3  # Prevent infinite loops
MAX_SEARCHES_PER_ITERATION = 5  # Control fanout

2. Budget Management

python
def query_with_budget(query, max_cost=0.50):
    spent = 0
    context = []
    
    while spent < max_cost:
        step_cost = estimate_cost(next_action)
        if spent + step_cost > max_cost:
            break
        
        # Execute and track
        result = execute_step(next_action)
        spent += step_cost
        context.append(result)
    
    return synthesize(context, query)

3. Fallback Strategy

python
def agentic_with_fallback(query, timeout=5.0):
    try:
        with timeout_context(timeout):
            return agentic_rag(query)
    except TimeoutError:
        # Fall back to standard RAG
        return simple_rag(query)

Related Papers

Next Steps

  1. ▸Try it: Run the Agentic RAG notebook
  2. ▸Combine: Use with Self-RAG (#14) for quality checks
  3. ▸Scale: Try Tree of Thoughts (#15) for complex reasoning
  4. ▸Deploy: Use Production RAG (#35) patterns

Part of: 37 RAG Patterns Collection
See also: Visual Architecture Guide

Related Concepts

Corrective RAG (CRAG) - Self-Correcting Retrieval

Automatically detect and correct irrelevant or incorrect retrievals. Uses relevance grading and web search fallback for improved accuracy.

Self-RAG

The smart decision maker — decides if retrieval is needed, filters irrelevant docs, generates multiple answers, and picks the best one.

Go Deeper With Live Instruction

Covered in depth in our llm engineering program.