Skip to content

ADR-008: Readiness Gate Policy and Failure Semantics

  • Decision status: Accepted
  • Implementation status: Not started
  • Applies to: Reference runtime, external targets
  • Date: 2026-06-16
  • Deciders: Grey Newell
  • Tags: release-control, policy, scorecard, gates, ci-cd, evals

Context

ADR-002 defines the ReadinessScorecard as the human/product release decision surface. ADR-007 requires scorecards and production evals to use the shared QueryPipeline. The next architectural question is how RAGhelm computes scorecard decisions such as ship, needs_review, or block.

A release-control product must be deterministic, explainable, auditable, and adaptable across projects. Hardcoded thresholds are fast but brittle. LLM-assisted summaries can help humans understand failures, but an LLM should not be the authority deciding whether a release ships.

This decision supports related implementation work without making issue numbers part of the durable architecture contract.

Decision

Use a versioned YAML/JSON release policy evaluated deterministically to compute ReadinessScorecard status.

The policy file defines:

  • required artifacts and freshness rules
  • production/local/demo mode rules
  • retrieval quality thresholds
  • generation quality thresholds
  • regression tolerance thresholds
  • latency and cost budgets
  • safety/human-review requirements
  • status mapping rules for ship, needs_review, and block

The deterministic policy evaluator is the source of truth for the scorecard decision. LLMs may summarize or explain the resulting checks, but they must not be the authoritative gate evaluator.

Minimum status semantics:

  • ship: all required gates pass, no unresolved review-required gates remain, and evidence is production-grade when making production claims.
  • needs_review: evidence is incomplete, local/demo-only, missing non-critical telemetry, under review threshold, or requires human approval for high-risk workflow categories.
  • block: a required artifact is missing, a critical metric fails, production mode is unsafe/misconfigured, manifest/provenance is missing, or a required security/safety/policy gate fails.

Consequences

Positive

  • Makes release decisions explainable and reproducible.
  • Allows different projects, demos, customer deployments, and workflows to use different thresholds without code changes.
  • Lets the policy version become part of RAGRunManifest provenance.
  • Supports CI/CD release gates because policy evaluation is deterministic.
  • Prevents LLM-generated rationales from silently changing release outcomes.

Negative

  • Requires schema design, validation, and policy loading.
  • Requires teams to maintain policy versions as product expectations evolve.
  • Misconfigured policies can block or permit releases incorrectly if not tested.

Neutral / Trade-offs

  • A default policy should exist for local/demo use, but it must be labeled as such.
  • Production policies should be stricter than demo policies and should fail closed when required evidence is missing.
  • Policy-as-code plugins may be added later, but the first production path should remain simple and deterministic.

Alternatives Considered

  1. Static thresholds hardcoded in code

Pros: - Fastest to implement. - Easy to understand for a small demo.

Cons: - Hard to adapt per project or customer. - Makes policy changes look like code changes rather than release-governance changes. - Weak provenance unless code SHA alone is treated as policy version.

  1. Versioned YAML/JSON release policy evaluated deterministically

Pros: - Explicit, auditable, configurable, and versionable. - Fits scorecards, manifests, CI gates, and customer-specific workflows. - Keeps release decisions deterministic.

Cons: - Requires policy schema and validation.

This is the chosen option.

  1. Policy-as-code plugin interface

Pros: - Most flexible for enterprise customization. - Could support complex organization-specific release rules.

Cons: - Too heavy for the current roadmap stage. - Harder to audit and sandbox.

  1. LLM/judge-assisted gate reasoning

Pros: - Can explain nuanced tradeoffs and produce user-friendly summaries.

Cons: - Not deterministic enough to be the source of release truth. - Adds cost, latency, and governance complexity to every gate.

Implementation Guidance

A policy should be stored as a versioned YAML or JSON artifact, for example:

schema_version: 1
policy_id: default-ragops-release-policy
version: 2026-06-16
mode: production
required_artifacts:
  manifest: block
  production_eval_summary: block
freshness:
  max_age_hours: 168
metrics:
  retrieval:
    recall@5:
      ship_min: 0.90
      review_min: 0.85
    mrr:
      ship_min: 0.75
      review_min: 0.70
    ndcg@5:
      ship_min: 0.80
      review_min: 0.75
  generation:
    faithfulness:
      ship_min: 4.0
      review_min: 3.0
regression:
  max_relative_drop: 0.05
operations:
  p95_latency_ms:
    ship_max: 2500
    review_max: 4000
  cost_per_successful_answer_usd:
    ship_max: 0.05
safety:
  high_risk_business_advice:
    unattended_release: needs_review

The evaluator should emit structured checks with:

  • check ID
  • label
  • status: pass, review, block
  • observed value
  • threshold or required condition
  • evidence source
  • remediation guidance

Policy evaluation requirements:

  • fail closed for missing required production artifacts
  • distinguish local/demo evidence from production evidence
  • include policy ID/version in the scorecard and RAGRunManifest
  • make status derivation machine-readable and testable
  • support targeted unit tests for every status mapping

Non-goals:

  • policy plugin runtime in the first implementation
  • LLM-controlled release decisions
  • enterprise workflow approvals before the scorecard policy evaluator exists

References

  • ADR-002: Release Readiness Scorecard and RAGRunManifest as Core Product Objects
  • ADR-007: Shared QueryPipeline as Production-Eval Source of Truth