Skip to content

ADR-011: Retrieval Routing, Hybrid Search, and Reranking Policy

  • Decision status: Accepted
  • Implementation status: Not started
  • Applies to: Reference runtime
  • Date: 2026-06-16
  • Deciders: Grey Newell
  • Tags: retrieval, routing, namespaces, hybrid-search, reranking, evals, cost

Context

RAGhelm's architecture includes multiple retrieval strategy capabilities: multi-strategy chunking, namespace routing, cross-namespace fusion, hybrid retrieval, reranking, retrieval experiments, and public-claim alignment.

ADR-005 establishes Pinecone namespaces as retrieval-strategy boundaries. ADR-007 requires served query behavior and production evals to share the same QueryPipeline. ADR-008 requires release decisions to come from deterministic policy evaluation. Retrieval routing, hybrid search, and reranking must therefore be explicit, versioned, measured, and traceable.

This decision supports related implementation work without making issue numbers part of the durable architecture contract.

Decision

Use a versioned retrieval policy with:

  1. a deterministic baseline router as the default trusted path
  2. optional evaluated LLM routing strategies
  3. conditional hybrid retrieval
  4. conditional fusion and reranking
  5. promotion to production only through measured eval evidence

The deterministic router is always available, auditable, and testable. LLM routing may be added as a policy-selected strategy only when evals prove quality lift and the cost/latency impact is acceptable.

Hybrid retrieval and reranking are not default taxes. They are conditionally enabled by retrieval policy based on route, query type, risk, exact lexical triggers, candidate confidence, and measured eval lift.

Policy Shape

A retrieval policy should define:

  • policy ID and version
  • available namespaces and their intended strategy
  • deterministic routing rules
  • optional LLM router configuration
  • exact lexical triggers for hybrid retrieval
  • cross-namespace fusion rules
  • reranking eligibility and skip policy
  • top-k and candidate budgets
  • cost and latency budgets
  • eval slices required before promotion

Example policy fields:

schema_version: 1
policy_id: default-ragops-retrieval-policy
version: 2026-06-16
default_route:
  namespace: default-512-small
  top_k: 5
routes:
  factual_lookup:
    namespace: semantic-256-large
    top_k: 5
    hybrid: conditional
    rerank: false
  exact_lexical_lookup:
    namespace: hybrid-1024-large
    top_k: 10
    hybrid: true
    rerank: conditional
  synthesis:
    namespace: longform-1024-large
    top_k: 8
    fusion: conditional
    rerank: conditional
router:
  baseline: deterministic
  llm_router: evaluated_optional
reranker:
  provider: cohere
  require_eval_lift: true
  max_p95_latency_ms_added: 500
  max_cost_usd_per_request_added: 0.01

Consequences

Positive

  • Keeps retrieval routing explainable and testable.
  • Prevents LLM routing, hybrid retrieval, and reranking from becoming unmeasured default costs.
  • Aligns namespace strategy with scorecards, manifests, and release policies.
  • Lets RAGhelm honestly downgrade README claims until a strategy is implemented and evaluated.
  • Supports experiment-driven promotion of chunking, embedding, hybrid, fusion, and reranking changes.

Negative

  • Requires a retrieval policy schema and evaluator.
  • Requires dedicated eval slices for exact lexical, factual, synthesis, and high-risk query categories.
  • Adds trace/provenance requirements for every retrieval decision.

Neutral / Trade-offs

  • The deterministic baseline router may be less adaptive than an LLM router, but it is safer as the default release-control path.
  • LLM routing remains available as an evaluated strategy rather than a foundational dependency.
  • Hybrid search and reranking can be powerful but must justify their incremental cost and latency.

Alternatives Considered

  1. Deterministic rule-based router only

Pros: - Transparent, cheap, and testable.

Cons: - May underperform on nuanced queries. - Limits future adaptive routing options.

  1. LLM classifier router as default

Pros: - More adaptive and expressive.

Cons: - Adds cost, latency, non-determinism, and failure modes. - Harder to treat as default release infrastructure without strong eval coverage.

  1. Versioned retrieval policy with deterministic baseline and optional evaluated LLM strategy

Pros: - Best balance of safety, extensibility, cost control, and eval-driven improvement. - Makes routing decisions auditable and manifestable.

Cons: - Requires policy schema and additional tests.

This is the chosen router approach.

  1. Always search all namespaces and fuse/rerank everything

Pros: - Simple to reason about as a brute-force recall strategy.

Cons: - Expensive, slower, and hides the routing/cost decisions RAGhelm should make explicit.

Hybrid Retrieval and Reranking Semantics

Hybrid retrieval should be used when exact lexical recall matters, including:

  • product names
  • customer names
  • policy IDs
  • ticket IDs
  • acronyms
  • error messages
  • contractual or exact phrase queries
  • feature names and proper nouns

Reranking should be used when:

  • candidate sets are noisy
  • top-3 precision materially affects answer quality
  • cross-namespace fusion produces mixed-quality results
  • high-risk workflows justify the incremental cost/latency
  • evals show NDCG/MRR lift on the relevant slice

Reranking should be skipped when:

  • exact sparse match confidently identifies the answer
  • the request is low-risk and latency-sensitive
  • cost budget is exhausted
  • evals do not show meaningful lift
  • production credentials/config for the reranker are unavailable

Required Observability and Manifest Fields

Every pipeline run should record:

  • retrieval policy ID/version
  • router implementation: deterministic or LLM
  • selected route/query type
  • candidate namespaces
  • selected namespace(s)
  • top-k per retrieval stage
  • hybrid enabled/disabled and reason
  • fusion enabled/disabled and reason
  • reranker enabled/disabled and reason
  • retrieved chunk IDs and scores
  • reranked chunk IDs and scores where applicable
  • retrieval latency by stage
  • incremental cost from LLM routing/reranking where applicable

Implementation Guidance

Initial implementation should:

  • introduce a versioned retrieval policy artifact
  • implement deterministic baseline routing first
  • make any LLM router explicitly optional and eval-gated
  • add hybrid/reranking only behind policy gates
  • add eval slices for exact lexical lookup, factual lookup, synthesis, and noisy candidates
  • include policy version and selected strategy in RAGRunManifest and scorecard evidence
  • downgrade public claims until the corresponding retrieval behavior is implemented and evaluated

Non-goals:

  • always-on LLM routing
  • always-on reranking
  • treating hybrid retrieval as a universal default
  • promoting retrieval strategies without before/after eval results

References

  • ADR-005: Pinecone Reference Backend Namespace Strategy
  • ADR-007: Shared QueryPipeline as Production-Eval Source of Truth
  • ADR-008: Readiness Gate Policy and Failure Semantics