Overview
| Property | Detail |
|---|---|
| Pattern | Self-RAG — The Smart Decision Maker |
| Level | 3 (Better retrieval decisions) |
| Key Innovation | Decides IF retrieval is needed, then filters + scores |
| LLM Calls | 5+ (assess, filter, generate multiple, score, select) |
| Notebook | 19_Self_RAG.ipynb |
Architecture
flowchart TD
A[User Query] --> B{"Do I need to retrieve docs?"}
B -->|No| C[Answer from Model Knowledge]
B -->|Yes| D[Retrieve from Qdrant]
D --> E{"Filter: Is each doc relevant?"}
E --> F["Doc 1: Relevant"]
E --> G["Doc 2: Relevant"]
E --> H["Doc 3: Irrelevant — Discard"]
F --> I[Generate 3 Candidate Answers]
G --> I
I --> J["Score Each: Utility 1-5"]
J --> K["Answer A: 3/5"]
J --> L["Answer B: 4/5 — WINNER"]
J --> M["Answer C: 3/5"]
L --> N[Return Best Answer]Test Results (Actual Notebook Output)
| Query | Retrieval Needed? | Docs Retrieved | Docs Relevant | Result |
|---|---|---|---|---|
| "Impact of climate change?" | No | — | — | Answered from model knowledge |
| "How did Harry beat Quirrell?" | Yes | 3 | 0 (all irrelevant) | Answered from knowledge |
| "International agreements?" | Yes | 3 | 3 (all relevant) | Generated 3 answers → picked best (utility 4/5) |
Decision Logic
| Condition | Action |
|---|---|
| Model already knows the answer | Skip retrieval (save time + cost) |
| Retrieved docs are irrelevant | Discard them, answer from knowledge |
| Retrieved docs are relevant | Generate multiple answers, score, return best |
Self-RAG vs Simple RAG
| Aspect | Simple RAG | Self-RAG |
|---|---|---|
| Retrieval | Always retrieves | Decides if needed |
| Relevance check | None | Filters each doc |
| Answers generated | 1 | Multiple (picks best) |
| Off-topic queries | Returns noise | Handles gracefully |
| LLM calls | 1 | 5+ |
| Cost | Low | Higher |
When to Use
| Scenario | Use Self-RAG? |
|---|---|
| Mixed query types (some need docs, some don't) | Yes |
| Questions that may be outside your corpus | Yes |
| Need highest single-answer quality | Yes |
| Latency-critical application | No — too many LLM calls |
| Simple factual lookup | No — overkill |
Tech Stack
| Component | Technology |
|---|---|
| Embeddings | AWS Bedrock — Amazon Titan Embed Text v2 (1024 dims) |
| LLM | AWS Bedrock — Claude Sonnet (cross-region inference) |
| Vector DB | Qdrant Cloud (cosine similarity) |
| PDF Reader | PyMuPDF (fitz) |
| Framework | LangChain |