Cross-Encoder Reranking - Precise Relevance Scoring

Overview

Cross-Encoder Reranking is a two-stage retrieval strategy that pairs a fast bi-encoder for initial candidate retrieval with a high-accuracy cross-encoder for precise relevance scoring. The cross-encoder reads query and document together, letting every token in the query attend to every token in the document — producing much richer relevance signals than bi-encoder embeddings alone.

Pattern #4 in the 37 RAG Patterns Collection

flowchart TD
  Q[Query] --> BI[Bi-encoder: embed query fast]
  BI --> RC["Recall stage: top-50 candidates from vector index"]
  RC --> CE{"Cross-encoder: jointly score each query+chunk pair — slower but accurate"}
  CE --> SR[Sort by cross-encoder score]
  SR --> TOP[Top-5 reranked results]
  TOP --> LLM[Generate]

The Problem

Bi-encoders encode query and document independently, then compare their embeddings with a dot product or cosine similarity:

Query:    encode("What causes inflation?")   → vector_q
Document: encode("Inflation is caused by…") → vector_d
Score:    cosine(vector_q, vector_d) = 0.72

This is fast — but the query and document never see each other during encoding, so subtle relevance signals are lost. A document that contains the exact right answer buried in a long paragraph may score lower than a shorter document full of topically related but non-answering content.

Query: "What is the penalty for late tax filing in Germany?"

Bi-encoder top result: "German tax system overview" (score 0.81)
  → Topically close but never mentions penalties

Actual best answer: "§152 AO — Late filing carries a surcharge of
  up to 10% of assessed tax." (score 0.74)
  → Buried in a statutory reference document, ranked #12

Solution

A cross-encoder reads [QUERY + DOCUMENT] as a single concatenated input, so every attention head can relate query tokens to document tokens directly:

Stage 1 — Bi-encoder retrieval (fast, ~10ms)
  Query → top-100 candidate documents

Stage 2 — Cross-encoder reranking (precise, ~200ms for 100 docs)
  [Query + Doc_1] → relevance score 0.94
  [Query + Doc_2] → relevance score 0.87
  …
  [Query + Doc_100] → relevance score 0.31
  → Sort, keep top-5

Implementation

python

from sentence_transformers import CrossEncoder

# Load cross-encoder model
# ms-marco-MiniLM-L-6-v2: fast (6-layer), good for production
# ms-marco-electra-base: slower, highest accuracy
cross_encoder = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")

def rerank(query: str, candidates: list[dict], top_k: int = 5) -> list[dict]:
    """
    Re-score a candidate list with a cross-encoder and return top_k results.
    candidates: list of {"content": str, "source": str, ...}
    """
    # Build (query, passage) pairs
    pairs = [(query, doc["content"]) for doc in candidates]

    # Cross-encoder scores — single forward pass over all pairs
    scores = cross_encoder.predict(pairs)

    # Attach scores and sort descending
    for doc, score in zip(candidates, scores):
        doc["rerank_score"] = float(score)

    ranked = sorted(candidates, key=lambda d: d["rerank_score"], reverse=True)
    return ranked[:top_k]


def cross_encoder_rag(query: str) -> tuple[str, dict]:
    # Stage 1: bi-encoder retrieval — broad recall
    candidates = vector_store.search(query, top_k=100)

    # Stage 2: cross-encoder reranking — precise relevance
    top_docs = rerank(query, candidates, top_k=5)

    # Generate answer from the reranked context
    context = "\n\n".join(doc["content"] for doc in top_docs)
    response = llm.generate(f"Context: {context}\n\nQuestion: {query}")

    return response, {
        "candidates_retrieved": len(candidates),
        "docs_after_rerank": len(top_docs),
        "top_score": top_docs[0]["rerank_score"] if top_docs else None,
    }

Choosing a Cross-Encoder Model

python

# Speed-optimised (6-layer MiniLM, ~2ms/pair on CPU)
fast_model = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")

# Accuracy-optimised (ELECTRA-base, ~8ms/pair on CPU)
accurate_model = CrossEncoder("cross-encoder/ms-marco-electra-base")

# Multilingual (mMiniLM, supports 100+ languages)
multilingual_model = CrossEncoder("cross-encoder/mmarco-mMiniLMv2-L12-H384-v1")

Complete Notebook

Run the full implementation:

Cross-Encoder Reranking AWS Notebook

Includes:

▸Bi-encoder vs cross-encoder latency benchmarks
▸Model comparison (MiniLM vs ELECTRA vs mMiniLM)
▸AWS Bedrock integration with Amazon Rerank
▸Batch scoring optimisation

Performance

Metric	Bi-Encoder Only	+ Cross-Encoder Rerank
Precision@5	62%	78%
MRR@10	0.61	0.79
Latency	~10ms	~210ms
Cost/query	$0.002	$0.006

Cross-encoder reranking adds ~200ms but raises Precision@5 by 16 percentage points.

When to Use

Best For

▸High-stakes search — legal, medical, financial queries where the top result must be correct
▸Small candidate sets — retrieving 50–200 candidates to rerank (not 10,000+)
▸Latency budget allows — sub-500ms total is achievable with MiniLM models
▸Domain-specific corpora — fine-tuned cross-encoders outperform general embeddings

Combine With

▸Fusion Retrieval (#3) — run RRF across multiple retrievers first, then rerank the merged candidates for both diverse recall and precise ranking
▸Contextual Compression (#6) — compress the top-5 reranked documents before passing to the LLM to cut token costs

Next Steps

▸Try it: Run the notebook
▸Upgrade recall: Combine with Fusion Retrieval (#3) to diversify candidates before reranking
▸Cut costs: Pipe reranked results through Contextual Compression (#6)
▸Go production: Fine-tune the cross-encoder on domain-specific query-passage pairs

Part of: 37 RAG Patterns Collection See also: Complete Pattern Index