Skip to content

ADR-010: Production Claims, Badges, and Public Proof Source of Truth

  • Decision status: Accepted
  • Implementation status: Not started
  • Applies to: Reference runtime, future
  • Date: 2026-06-16
  • Deciders: Grey Newell
  • Tags: badges, public-claims, scorecard, manifest, ci-cd, proof-artifacts

Context

RAGhelm's README badges, public quality claims, proof artifacts, and release-readiness demos are credibility surfaces. Because RAGhelm's product spine is release control, the project must not publish quality numbers that come from local, mock, stale, incomplete, or unaudited artifacts.

ADR-002 defines the ReadinessScorecard and RAGRunManifest as core product objects. ADR-008 defines deterministic release policies. ADR-009 requires versioned manifests with artifact hashes and runtime metadata.

The remaining question is which artifacts are allowed to back public claims and generated badges.

This decision supports public-safe demo/proof artifacts, milestone gates, and claim-retirement criteria without making issue numbers part of the durable architecture contract.

Decision

Use two linked rules:

  1. Semantic source of truth: public quality claims must be backed by a ReadinessScorecard linked to a valid production RAGRunManifest.
  2. Published badge source of truth: README badges and other published badge assets must be generated from a CI/release artifact bundle containing the valid production ReadinessScorecard and linked RAGRunManifest.

A public claim is valid only when:

  • the scorecard exists and has a recognized schema version
  • the scorecard links to a valid RAGRunManifest
  • the manifest mode is production
  • the release policy ID/version is present
  • required production eval artifacts are present and hashed
  • release-critical artifact hashes validate
  • the artifact is fresh enough for the claim being made
  • privacy classification permits public proof publication
  • the scorecard status is compatible with the claim text

If those conditions are not met, RAGhelm must fail closed. Badges and public copy should show unavailable, stale, local-demo, or needs-review rather than presenting unsupported quality scores.

Consequences

Positive

  • Protects RAGhelm's credibility as a release-control product.
  • Prevents local/demo/mock evals from becoming public production claims.
  • Makes public-facing proof artifacts auditable and reproducible.
  • Aligns badges, README claims, scorecards, manifests, and CI/CD release bundles.
  • Gives implementation agents clear rules for badge generation and docs updates.

Negative

  • Requires CI/release workflows to preserve and validate scorecard/manifest bundles.
  • Badges may show unavailable/stale until production evidence exists.
  • Slightly slows marketing/docs updates because claims must be backed by artifacts.

Neutral / Trade-offs

  • Local/demo scorecards remain useful for demos, but they must be labeled as local/demo evidence.
  • A needs_review scorecard may be shown publicly only if the copy explicitly says the release needs review and does not imply a production pass.
  • CI becomes the publication boundary for badges, while scorecard+manifest remains the semantic product truth.

Alternatives Considered

  1. Any latest eval summary

Pros: - Simplest badge generation path. - Easy to implement from current artifacts.

Cons: - Unsafe because local/demo/mock outputs can become public claims. - Lacks release decision and provenance.

  1. Production eval summary only

Pros: - Better than accepting local/demo outputs. - Simple to reason about.

Cons: - Still lacks full scorecard decision semantics. - Does not require linked manifest provenance or policy version.

  1. Scorecard with linked production RAGRunManifest only

Pros: - Strong semantic source of truth. - Links claims to both release decision and provenance. - Fits ADR-002 and ADR-009.

Cons: - Requires implementing scorecard/manifest validation.

This is the chosen semantic rule.

  1. CI-generated release artifact only

Pros: - Strongest discipline for published badges. - Ensures public assets come from a reproducible workflow.

Cons: - Requires CI wiring and artifact preservation. - Slightly stricter than the semantic rule alone.

This is the chosen published-badge rule.

Implementation Guidance

Badge and public-proof generation should validate:

  • scorecard schema version
  • scorecard status
  • manifest schema version
  • manifest mode equals production for production claims
  • release policy ID/version exists
  • required artifact hashes exist and match
  • eval summary source is production
  • artifact freshness is within policy
  • privacy flags allow public publication

Badge states should include more than pass/fail:

  • passing or metric value when backed by a valid production scorecard and manifest
  • needs_review when evidence exists but the scorecard requires review and the claim text says so
  • stale when evidence is too old
  • unavailable when required production evidence is missing
  • local_demo when only local/demo evidence exists
  • blocked when the scorecard blocks release

README/public documentation rules:

  • Do not claim production quality from local/demo artifacts.
  • Prefer stale-but-real production evidence over fresh fake/demo evidence.
  • Link badges to the backing scorecard or CI release artifact bundle, not only raw eval summaries.
  • Include manifest and policy IDs in proof artifacts when practical.

Non-goals:

  • onchain attestation as a prerequisite for public badges
  • signed manifests before the manifest schema stabilizes
  • hiding needs_review or unavailable states to make the project look healthier

References

  • ADR-002: Release Readiness Scorecard and RAGRunManifest as Core Product Objects
  • ADR-008: Readiness Gate Policy and Failure Semantics
  • ADR-009: RAGRunManifest Schema and Artifact Lifecycle