Production RAG Evaluation: Measure Retrieval Before You Ship

[Definition] A production RAG evaluation suite measures two separate things: whether the system retrieved the evidence it needed, and whether the final answer used that evidence faithfully. A fluent answer is not proof that either happened.

The RAG quality chain

Question
  ↓
Query rewrite / routing
  ↓
Retrieval → reranking → context assembly
  ↓
LLM answer with citations
  ↓
User outcome

A failure at any stage can look like the same user complaint: “the answer was wrong.” Evaluation makes the stage visible.

Start with a golden dataset

A golden dataset is a small, versioned set of questions with expected evidence and, where possible, expected answer properties. It is more valuable than a large unlabelled log export.

{
  "id": "policy-014",
  "question": "What is the cancellation window for enterprise contracts?",
  "relevant_chunk_ids": ["msa-12", "terms-4"],
  "relevance": {"msa-12": 3, "terms-4": 2},
  "must_cite": ["msa-12"],
  "answer_notes": "Must distinguish enterprise from self-service terms"
}

Build the first dataset from real high-value questions:

1. ▸queries that support teams answer repeatedly;
2. ▸queries that previously caused hallucinations or escalations;
3. ▸ambiguous questions requiring a safe “I don’t know”;
4. ▸multi-hop questions requiring two or more sources;
5. ▸edge cases across languages, dates, document versions, and permissions.

Do not let an LLM generate the only labels. It can accelerate drafting, but a domain owner should verify evidence relevance and the unacceptable-answer cases.

Retrieval metrics

Recall@k: did we retrieve the evidence?

Recall@k = relevant documents retrieved in top k / all relevant documents

If a question has two required chunks and top-10 contains one, Recall@10 is 0.5. For RAG, this is often the first metric to investigate: generation cannot cite evidence that retrieval missed.

MRR: how early is the first useful result?

MRR = average(1 / rank of first relevant result)

MRR rewards a correct document near the top. It is useful for single-answer questions but insufficient for multi-source answers.

NDCG@k: are the best sources ranked highest?

NDCG supports graded relevance: a definitive policy clause can score 3, a related summary 2, and a weak mention 1. It discounts relevant material that appears lower in the list.

[Key Insight] Run exact retrieval on a representative sample to create ground truth. Approximate search can then be compared against that baseline while tuning HNSW, filters, rerankers, and chunking.

Context and answer metrics

Retrieval metrics stop at the context boundary. RAG needs additional measures after the LLM sees that context.

Context precision

Of the chunks supplied to the model, how many are actually useful for the question? Low precision wastes context window, raises cost, and increases the chance of distracting the model.

Context recall

Does the supplied context contain the information needed to answer? A response can be faithful to its context yet incomplete because an important chunk was missing.

Faithfulness / groundedness

Does every factual claim in the answer follow from the retrieved context? A judge can break an answer into claims and mark each as supported, contradicted, or unsupported.

faithfulness = supported answer claims / all verifiable answer claims

Answer relevance

Does the answer actually address the user’s question? A perfectly cited answer that answers a neighbouring question still fails the user.

Using an LLM-as-judge safely

LLM judges are useful for scale, but treat them as measurement instruments that require calibration:

• ▸give the judge only the question, context, answer, and a precise rubric;
• ▸request structured output with per-claim evidence;
• ▸compare judge scores with human labels on a holdout set;
• ▸freeze the judge model and prompt version for a benchmark run;
• ▸sample disagreements and low-confidence cases for review.

A judge should not decide policy correctness by itself. It can flag candidates; accountable domain review remains necessary for high-risk workflows.

A practical release gate

Define thresholds before changing an embedding model, chunk strategy, reranker, prompt, or index setting.

Release only if:
  Recall@10             ≥ 0.92
  NDCG@10               ≥ 0.85
  Faithfulness          ≥ 0.95
  Unsupported-claim rate ≤ 0.03
  P95 end-to-end latency ≤ 2.0 s
  No critical golden-case regression

The values are examples, not universal targets. A legal research system may require higher recall and abstention; a product search system may accept lower recall for a faster response. The important part is an explicit, versioned contract.

Evaluate slices, not only averages

Averages hide the failures that matter. Report metrics by slice:

• ▸question class: factual, comparison, multi-hop, summary, refusal;
• ▸document age and version;
• ▸language or business region;
• ▸tenant, access-policy, and permission boundary;
• ▸short versus long context;
• ▸head versus tail query frequency.

A model can improve the global average while breaking a high-value compliance slice.

Online monitoring after release

Offline evaluation decides whether a change is safe to launch. Online signals show whether it remains useful:

Store the model, embedding model, index version, prompt version, retrieval parameters, sources, latency, and answer outcome together in one trace. Without that lineage, an observed regression is difficult to reproduce.

Free concepts, Pro execution

This guide is public because every team should understand what good RAG evidence looks like. The Pro production path applies the same framework to a learner’s own corpus: define a golden set, compare retrieval configurations, inspect groundedness failures, and decide a release threshold. A Live Cohort capstone adds review of the evaluation design and deployed dashboard.

Next steps

1. ▸Read Measuring Search Quality for retrieval metrics.
2. ▸Read Grounding for claim-level evidence checks.
3. ▸Read Reliable RAG for verification layers.
4. ▸Use the learning roadmap to see how evaluation fits the Pro production path.