Overview
| Property | Detail |
|---|---|
| Pattern | Retrieval with Feedback Loop — The Learning System |
| Level | 6 (Continuous improvement) |
| Key Innovation | Past feedback re-ranks future retrievals + adds good Q&A pairs to the index |
| Notebook | 19_Self_RAG.ipynb |
Architecture
flowchart TD
A[User Query] --> B[Retrieve from Qdrant]
B --> C["Adjust Scores Using Past Feedback\n(Boost high-rated, demote low-rated)"]
C --> D["Generate Answer (LLM via Bedrock)"]
D --> E["User Rates: 1-5"]
E --> F{"Score >= 4?"}
F -->|Yes| G["Add Q&A Pair to Qdrant Index"]
F -->|No| H[Store Feedback Only]
G --> I["System is Now SMARTER"]
H --> I
I -->|Next Query| ATest Results (Actual Notebook Output)
| Step | What Happened |
|---|---|
| Query 1 | "What is the greenhouse effect?" → Retrieved docs → Generated answer |
| User rates | 5/5 (great answer) |
| Feedback stored | Query + docs + score saved |
| Index grows | Collection: 92 → 93 points (added the Q&A pair) |
| Query 2 | "How do gases trap heat?" |
| Result | The feedback-derived Q&A pair appeared as top result! |
The system literally learned from its own good answers.
How Feedback Improves Retrieval
| Mechanism | How It Works |
|---|---|
| Score boosting | Docs that got 5/5 in past → boosted in future rankings |
| Score demotion | Docs that got 1/5 in past → demoted in rankings |
| Index expansion | High-quality Q&A pairs added to Qdrant as new documents |
| Query memory | "This query + these docs = good result" is remembered |
When to Use
| Scenario | Use Feedback Loop? |
|---|---|
| Repeated similar queries over time | Yes — learns what works |
| Internal tool used by same team | Yes — adapts to team's needs |
| Public-facing with diverse queries | Maybe — feedback may be noisy |
| One-shot queries (no repeat users) | No — no opportunity to learn |
Tech Stack
| Component | Technology |
|---|---|
| Embeddings | AWS Bedrock — Amazon Titan Embed Text v2 (1024 dims) |
| LLM | AWS Bedrock — Claude Sonnet |
| Vector DB | Qdrant Cloud (cosine similarity) |
| Feedback Store | In-memory (upgradeable to persistent) |
| Framework | LangChain |