Skip to content

ADR-007: Shared QueryPipeline as Production-Eval Source of Truth

  • Decision status: Accepted
  • Implementation status: Not started
  • Applies to: Reference runtime
  • Date: 2026-06-16
  • Deciders: Grey Newell
  • Tags: query-pipeline, production-eval, release-control, scorecard, rag-pipeline

Context

ADR-001 positions RAGhelm as private RAGOps and release-control infrastructure. ADR-002 makes the ReadinessScorecard and RAGRunManifest the core product objects. ADR-006 defines a split runtime boundary where Python owns AI runtime behavior and TypeScript owns the product gateway.

For release control to be trustworthy, production evals must test the same behavior that will actually serve users. If the served query path and eval path diverge, a scorecard may pass behavior that is not deployed, or block behavior that users will never encounter.

This decision supports grounded generation, retrieval strategy routing, production source-of-truth rules, dashboard release-control surfaces, retrieval experiments, readiness scorecards, release gates, and proof artifacts.

Decision

Use one shared QueryPipeline as the source of truth for served query behavior, production evals, readiness scorecards, and release gates.

This applies to RAGhelm-owned reference runtime behavior. Black-box or external systems under test are evaluated through the TargetAdapter boundary from ADR-013.

Boundary rule:

  • QueryPipeline is canonical for RAGhelm-owned reference runtime behavior.
  • TargetAdapter is canonical for black-box or external systems under test.
  • Both boundaries must normalize run evidence into RAGRunManifest and ReadinessScorecard artifacts.

The QueryPipeline should orchestrate:

query/task input
  -> request validation and run context
  -> intent/risk/complexity classification
  -> retrieval routing and namespace/index selection
  -> retrieval and optional fusion/reranking
  -> context assembly and citation packing
  -> model/provider routing
  -> grounded generation or no-answer/escalation behavior
  -> trace/event capture
  -> response artifact usable by evals and scorecards

The same pipeline must be callable from:

  • Python FastAPI served query endpoints
  • production eval runner
  • scorecard/release-check generation
  • future async worker jobs
  • future TypeScript gateway calls into the Python runtime

Local/demo/production behavior should be selected through explicit configuration and dependency injection, not by maintaining separate orchestration logic.

Consequences

Positive

  • Makes readiness scorecards credible because they evaluate the same path that serves users.
  • Reduces duplicated routing, retrieval, generation, and trace logic.
  • Makes regressions easier to diagnose because eval artifacts and served traces share pipeline fields.
  • Supports ADR-002 by letting scorecards link directly to pipeline runs and manifests.
  • Provides a clean target for future TypeScript gateway contracts from ADR-006.

Negative

  • Requires refactoring if served API and eval runner currently have separate orchestration paths.
  • The pipeline interface must be carefully designed so local, demo, and production modes remain explicit.
  • Test coverage must include pipeline contract tests, not only endpoint or eval-runner tests.

Neutral / Trade-offs

  • The shared pipeline does not require every feature to be implemented immediately. Hybrid retrieval, reranking, agent steps, and external target adapters can be pipeline stages added behind explicit policies.
  • Local/mock mode remains valuable for development, but it must be clearly labeled and must not back production claims.
  • Provider implementations can vary by mode, but the orchestration contract should remain stable.

Alternatives Considered

  1. One shared QueryPipeline for served API and production eval

Pros: - Strongest release-control semantics. - Keeps evals, traces, scorecards, and served behavior aligned. - Reduces drift and duplicate implementation.

Cons: - Requires interface design and migration work.

This is the chosen option.

  1. Separate eval and served paths reconciled by artifact comparison

Pros: - Easier initially if current code paths differ. - Allows eval-specific shortcuts.

Cons: - Weakens trust in release gates. - Makes scorecards less defensible because they may not reflect deployed behavior.

  1. Shared interface with separate implementations

Pros: - Allows some implementation flexibility. - Better than fully separate paths.

Cons: - Still risks behavior drift unless contract tests are very strict. - More complex than a single orchestration path.

  1. Keep current split until later

Pros: - Lowest immediate implementation effort.

Cons: - Contradicts the release-control product spine. - Allows future issues to build on inconsistent behavior.

Implementation Guidance

The shared QueryPipeline should expose a typed input/output contract.

Minimum input fields:

  • query or task text
  • target identifier
  • mode: local, demo, staging, production
  • eval/run context, when invoked from release checks
  • optional policy overrides
  • trace/run ID

Minimum output fields:

  • answer or no-answer/escalation result
  • retrieved chunk IDs, scores, source docs, and citations
  • selected index/namespace/retrieval strategy
  • model/provider/judge/router choices
  • latency spans
  • token and cost estimates
  • failure mode, if any
  • trace ID, run ID, manifest ID, and scorecard linkage where applicable

Implementation requirements:

  • Served /query and production eval must call the same pipeline object/function.
  • Pipeline mode must be explicit; do not silently fall back from production to local/demo behavior.
  • Local/demo outputs must be labeled so they cannot back production claims.
  • Add tests proving served API and production eval use the same pipeline contract.
  • Add trace fields required by scorecards and manifests before public release claims depend on them.

Non-goals:

  • implementing every retrieval strategy before the pipeline exists
  • forcing the TypeScript gateway to duplicate pipeline logic
  • making local/mock evals equivalent to production release checks

References

  • ADR-001: Product Architecture Spine for Private RAGOps Release Control
  • ADR-002: Release Readiness Scorecard and RAGRunManifest as Core Product Objects
  • ADR-006: Canonical Runtime and API Boundary