Skip to content

ADR-009: RAGRunManifest Schema and Artifact Lifecycle

  • Decision status: Accepted
  • Implementation status: Partial
  • Applies to: Reference runtime, external targets
  • Date: 2026-06-16
  • Deciders: Grey Newell
  • Tags: manifest, provenance, artifacts, hashes, release-control, auditability

Context

ADR-002 defines the RAGRunManifest as the machine/audit provenance artifact linked to the human-facing ReadinessScorecard. ADR-008 requires release policies to evaluate required artifacts, freshness, production/local/demo mode, and evidence provenance deterministically.

To make scorecards trustworthy, RAGhelm needs a stable manifest standard that records what ran and what artifacts were produced. A minimal demo JSON is not enough for production release control. A full signed provenance bundle is attractive, but too heavy before the core schema is proven.

This decision supports related implementation work without making issue numbers part of the durable architecture contract.

Decision

Use a versioned RAGRunManifest schema with required artifact hashes and git/runtime metadata.

A RAGRunManifest is required for any scorecard that backs production release decisions or public quality claims. The manifest must identify:

  • schema and manifest identity
  • run identity and timestamps
  • source repository and git SHA
  • target under test
  • runtime mode: local, demo, staging, production
  • shared QueryPipeline or external target contract version
  • retrieval, generation, judge, and policy configuration
  • eval suite and release policy version
  • artifact paths and hashes
  • privacy/data classification
  • links to scorecards, traces, and eval summaries where available

Manifest hashes are required for release-critical artifacts such as eval summaries, scorecards, trace bundles, policy files, and benchmark proof outputs. Signing and external-standard compatibility are future extensions after the schema stabilizes.

Consequences

Positive

  • Gives scorecards an auditable provenance surface.
  • Supports CI/CD release gates and public proof artifacts.
  • Makes local/demo/staging/production evidence distinguishable.
  • Enables future onchain attestation by anchoring manifest hashes without exposing private data.
  • Supports external target adapters because all targets can normalize evidence into a common manifest shape.
  • Makes artifact drift and stale public claims easier to detect.

Negative

  • Requires schema validation and artifact hashing in release/eval workflows.
  • Requires careful privacy classification so manifests do not leak private customer data.
  • Requires migration discipline when manifest schema versions change.

Neutral / Trade-offs

  • The initial schema should be strict about required release-control fields but not try to model every future integration.
  • Manifest signing is deferred until the schema and artifact lifecycle prove useful.
  • OpenTelemetry/OpenInference compatibility should map into or out of the manifest rather than forcing the first schema to be fully external-standard-native.

Alternatives Considered

  1. Minimal local JSON schema only

Pros: - Fastest to implement. - Good enough for a local demo.

Cons: - Weak production auditability. - Does not sufficiently support public proof, CI gates, external adapters, or future attestation.

  1. Versioned schema with required artifact hashes and git/runtime metadata

Pros: - Strong enough for production release control. - Still implementable without signing infrastructure. - Supports scorecards, CI, benchmark proof, and future attestation.

Cons: - Requires schema validation and lifecycle management.

This is the chosen option.

  1. Full signed provenance bundle

Pros: - Strong enterprise/audit story. - Can prove artifact integrity and signer identity.

Cons: - Requires signing keys, verification, rotation, and trust-chain management. - Premature before the manifest schema stabilizes.

  1. External standard compatibility from day one

Pros: - Could align with OpenTelemetry, OpenInference, SLSA, or other provenance standards.

Cons: - May over-constrain RAGhelm before its product object is proven. - Adds mapping and dependency burden before the core release-control workflow is stable.

Required Schema Shape

Minimum manifest fields:

schema_version: 1
manifest_id: ragrun-<stable-run-id>
run_id: <stable-run-id>
created_at: <iso8601>
completed_at: <iso8601|null>
mode: local | demo | staging | production
release_candidate: <name-or-version>
source_control:
  repository: greynewell/raghelm
  git_sha: <sha>
target:
  id: <target-id>
  name: <human-name>
  type: reference_rag | reference_agent | external_rag | external_agent | api_target
  version: <optional-target-version>
pipeline:
  query_pipeline_version: <version-or-git-sha>
  retrieval_policy_version: <version|null>
  release_policy_id: <policy-id>
  release_policy_version: <policy-version>
models:
  embeddings:
    provider: <provider>
    model: <model>
    dimensions: <int>
  generation:
    provider: <provider>
    model: <model>
  judge:
    provider: <provider-or-offline>
    model: <model-or-rubric>
retrieval:
  vector_backend: <backend>
  index: <index|null>
  namespaces: [<namespace>]
  top_k: <int>
eval:
  suite: <suite-name>
  mode: local | demo | staging | production
  judge_mode: offline_heuristic | production | none
artifacts:
  - kind: eval_summary | scorecard | trace_bundle | policy | benchmark_report
    path: <path-or-uri>
    sha256: <required-for-release-critical-artifacts>
privacy:
  contains_private_customer_data: <bool>
  synthetic_demo_data: <bool>
  public_safe: <bool>
links:
  scorecard_id: <id|null>
  trace_ids: [<trace-id>]

Artifact Lifecycle Rules

  • Every manifest must be immutable once published as release evidence.
  • Corrections require a new manifest version or new run ID.
  • Release-critical artifacts must include SHA-256 hashes.
  • Public proof artifacts must not include private source content, credentials, or sensitive traces.
  • Production scorecards must fail closed when the linked manifest is missing, stale, has invalid schema, or lacks required hashes.
  • Local/demo manifests must be visibly labeled and must not back production badges or public quality claims.
  • Manifest schema changes require compatibility notes and migration guidance.

Implementation Guidance

Initial implementation should:

  • define JSON schema or Pydantic models for RAGRunManifest
  • add a helper to compute artifact SHA-256 hashes
  • emit manifests from production eval/release-check runs
  • link scorecards to manifest IDs and paths
  • validate manifest mode and privacy fields before public proof generation
  • add tests for missing hashes, invalid modes, stale artifacts, and schema-version handling

Non-goals:

  • signed manifests in the first implementation
  • onchain attestation in the first implementation
  • requiring full OpenTelemetry/OpenInference compatibility before the manifest schema is useful
  • storing private source content inside manifests

References

  • ADR-002: Release Readiness Scorecard and RAGRunManifest as Core Product Objects
  • ADR-008: Readiness Gate Policy and Failure Semantics