Skip to content

ADR-002: Release Readiness Scorecard and RAGRunManifest as Core Product Objects

  • Decision status: Accepted
  • Implementation status: Partial
  • Applies to: Reference runtime, external targets
  • Date: 2026-06-16
  • Deciders: Grey Newell
  • Tags: release-control, scorecard, provenance, evals, observability, ragops

Context

ADR-001 establishes RAGhelm's product architecture spine as private RAGOps and release-control infrastructure, demonstrated through a synthetic business-agent reference workflow.

That product spine needs concrete product objects that can connect evaluation, provenance, release gates, public-safe proof artifacts, trace interoperability, and dashboard workflows.

A plain eval summary is useful, but it does not answer the product question: can this private RAG or agent release ship? A provenance manifest is auditable, but by itself it does not communicate release readiness to a human operator or CI/CD gate.

RAGhelm needs linked artifacts that separate two concerns:

  1. what exactly ran and produced the evidence
  2. whether that evidence is sufficient to ship

Decision

RAGhelm's core product object is a linked pair:

  1. RAGRunManifest — the machine/audit provenance artifact that records what ran.
  2. ReadinessScorecard — the human/product release-control artifact that decides ship / block / needs review.

The RAGRunManifest answers:

What target, corpus, pipeline, models, prompts, eval suite, git SHA, environment, and artifacts produced this result?

The ReadinessScorecard answers:

Can this RAG or agent release ship, and why or why not?

The scorecard must link back to its manifest. The manifest must link to the eval and trace artifacts it summarizes. Public claims, dashboard cards, CI gates, and stakeholder proof should use the scorecard as the decision surface and the manifest as the audit surface.

Minimum release status values:

  • ship
  • needs_review
  • block

Minimum scorecard dimensions:

  • quality gates: retrieval and generation metrics
  • regression gates: deltas from baseline where available
  • operations gates: latency, cost, and freshness
  • provenance gates: manifest presence, production/local mode, source artifact hashes
  • safety/review gates: human review requirements for high-risk agent workflows

Thin first-slice build rule:

  • Implement a minimal schema and local demo surface first.
  • Do not require full OpenTelemetry import/export, external target adapters, onchain attestation, or enterprise policy workflows before the scorecard/manifest pair is useful.

Consequences

Positive

  • Makes RAGhelm's differentiated product object clear and demoable.
  • Turns evals from reports into release-control decisions.
  • Separates human decision UX from machine-auditable provenance.
  • Gives CI/CD, dashboard, benchmark, badge, and public-proof artifacts one consistent source of truth.
  • Supports future external target adapters because any RAG/agent stack can produce a manifest and scorecard.
  • Supports future optional attestation because manifest hashes can be anchored without exposing private RAG data.

Negative

  • Adds schema and artifact lifecycle work before some feature work can be considered production-ready.
  • Requires discipline to avoid treating local/mock scorecards as production proof.
  • Requires dashboard and docs updates so users understand scorecards are release decisions, not just visual summaries.

Neutral / Trade-offs

  • Existing eval summaries remain useful as input artifacts, but they are no longer the primary product object.
  • Production badges should point to scorecard-backed production summaries or fail closed when those artifacts are missing/stale.
  • RAGRunManifest fields can start minimal and become stricter as external adapters, OpenTelemetry/OpenInference, and BYOC deployment mature.

Alternatives Considered

  1. Eval summary only

Pros: - Fastest to build from current eval artifacts. - Simple to expose through CI and badges.

Cons: - Too generic and not a strong product wedge. - Does not naturally encode ship/block/needs-review decisions. - Weak provenance and audit story.

  1. Readiness scorecard only

Pros: - Strong human-facing demo and release-control object. - Easy to explain to users and operators.

Cons: - Without a manifest, users cannot easily audit what produced the score. - Risk of scorecards feeling arbitrary or disconnected from reproducible artifacts.

  1. RAGRunManifest only

Pros: - Strong provenance and reproducibility story. - Natural foundation for future attestation and external adapters.

Cons: - Does not directly answer whether a release can ship. - Less compelling as a demo surface by itself.

  1. Linked RAGRunManifest + ReadinessScorecard

Pros: - Strongest release-control architecture. - Human decision and machine provenance each have a clear role. - Aligns with the private RAGOps spine from ADR-001.

Cons: - Slightly more schema and UI work than a scorecard alone.

This is the chosen option.

Implementation Guidance

Initial implementation should:

  • define minimal scorecard and manifest schemas
  • emit or provide demo artifacts for a local release-readiness page
  • make the dashboard lead with release status rather than raw charts
  • show failed checks explicitly with remediation guidance
  • fail closed when production summaries or manifests are missing/stale
  • keep local/demo artifacts clearly labeled so they are not mistaken for production proof

Non-goals for the first implementation slice:

  • onchain attestation
  • full OpenTelemetry/OpenInference import/export
  • external target adapter framework
  • complete policy engine
  • enterprise audit store
  • full agent safety and human-review workflow

References

  • ADR-001: Product Architecture Spine for Private RAGOps Release Control