Skip to content

ADR-014: Native JSON Observability and Future OpenTelemetry Interoperability

  • Decision status: Accepted
  • Implementation status: Partial
  • Applies to: Reference runtime, external targets, future
  • Date: 2026-06-16
  • Deciders: Grey Newell
  • Tags: observability, json-artifacts, traces, opentelemetry, openinference, roadmap

Context

RAGhelm's product spine is release control for private RAG and agent systems. ADR-002 defines ReadinessScorecards and RAGRunManifests as core product objects. ADR-009 defines manifest provenance and artifact lifecycle. ADR-013 prioritizes external target adapters before internal vector backend abstraction.

The architecture includes observability, anomaly detection, cost/latency tracking, persisted run history, business outcome instrumentation, and OpenTelemetry/OpenInference interoperability.

The architectural question is what observability format should be the initial source of truth while the product object stabilizes.

This decision supports related implementation work without making issue numbers part of the durable architecture contract.

Decision

Use native JSON observability artifacts as the initial source of truth, and add OpenTelemetry/OpenInference interoperability as a later roadmap milestone.

Initial source of truth:

  • JSON ReadinessScorecards
  • JSON RAGRunManifests
  • JSON eval summaries and detailed eval runs
  • JSON trace/event bundles emitted by the shared QueryPipeline
  • JSON release policy and retrieval policy artifacts

OpenTelemetry/OpenInference support is explicitly deferred, not rejected. It should be implemented after the native scorecard, manifest, policy, and target-adapter contracts stabilize.

Consequences

Positive

  • Keeps the initial implementation simple and demo-friendly.
  • Lets RAGhelm stabilize its own release-control objects before mapping to external standards.
  • Works well with local demos, CI artifacts, GitHub release artifacts, and static dashboard previews.
  • Avoids premature dependency on OpenTelemetry/OpenInference schema choices before RAGhelm's product model is proven.
  • Keeps public proof artifacts easy to inspect in a repository or artifact bundle.

Negative

  • Native JSON alone is weaker for enterprise observability ecosystem integration.
  • External trace import/export will remain a roadmap item rather than immediate capability.
  • RAGhelm may need migration/mapping work later when adding OpenTelemetry/OpenInference support.

Neutral / Trade-offs

  • Native JSON artifacts should still use stable IDs and fields that can later map to OpenTelemetry/OpenInference spans and attributes.
  • RAGhelm should avoid becoming a generic trace viewer; OpenTelemetry/OpenInference support should serve release-control evidence, not replace specialized observability tools.
  • Prometheus-style operational metrics can still exist for service health, but release evidence should remain artifact-backed.

Alternatives Considered

  1. Native JSON artifacts only, indefinitely

Pros: - Simple and repository-friendly. - Easy to validate in CI.

Cons: - Weak enterprise interoperability. - Makes it harder to ingest traces from existing LLM observability stacks.

  1. Native database/store as immediate source of truth

Pros: - Stronger dashboard and historical querying story. - Enables comparisons and anomaly detection at scale.

Cons: - Adds persistence design before native artifact contracts are stable. - Less convenient for local demos and CI artifact proof.

  1. OpenTelemetry/OpenInference-native from the start

Pros: - Strong interoperability. - Good enterprise story.

Cons: - Over-constrains the initial design. - Risks making RAGhelm look like a generic observability tool rather than a release-control product.

  1. Native JSON now, OpenTelemetry/OpenInference later

Pros: - Best initial simplicity. - Preserves future interoperability. - Lets scorecards/manifests stabilize first.

Cons: - Requires an explicit future mapping milestone.

This is the chosen option.

Required Native JSON Event Fields

Native QueryPipeline trace/event bundles should include, where applicable:

  • schema_version
  • trace_id
  • run_id
  • target_id
  • scorecard_id
  • manifest_id
  • timestamp
  • mode: local, demo, staging, production
  • workflow or query_type
  • intent_class
  • risk_level
  • retrieval_policy_id and version
  • selected_index
  • selected_namespaces
  • retrieved_chunk_ids
  • reranker_used
  • model_provider, model_name, and model_role
  • prompt_version or rubric_version
  • input_tokens and output_tokens
  • estimated_cost_usd
  • latency_ms_by_span
  • quality_scores
  • failure_mode
  • external_trace_refs, when imported from another system

OpenTelemetry/OpenInference Roadmap Milestone

Add a later roadmap milestone for interoperability after native artifacts stabilize:

Milestone: OpenTelemetry/OpenInference trace import/export

Acceptance criteria:

  • map native RAGhelm trace/event fields to OpenTelemetry/OpenInference-compatible spans or attributes
  • import external traces and associate them with RAGRunManifest and ReadinessScorecard IDs
  • export RAGhelm pipeline spans without leaking private content by default
  • preserve trace IDs across TypeScript gateway, Python runtime, eval runs, scorecards, and manifests
  • document which fields are required, optional, redacted, or unsupported
  • add tests for round-trip mapping where feasible

Implementation Guidance

Initial implementation should:

  • keep scorecards, manifests, policies, eval summaries, and trace bundles as versioned JSON artifacts
  • add schema validation for release-critical artifacts
  • preserve stable IDs so later OTel/OpenInference mapping is straightforward
  • keep private source content out of public JSON proof artifacts
  • use native JSON as CI/release artifact source of truth per ADR-010

Non-goals:

  • immediate OpenTelemetry/OpenInference implementation
  • replacing Langfuse, Phoenix, LangSmith, or similar trace viewers
  • building a database-backed observability store before native artifact contracts are stable
  • exporting private retrieved content by default

References

  • ADR-002: Release Readiness Scorecard and RAGRunManifest as Core Product Objects
  • ADR-009: RAGRunManifest Schema and Artifact Lifecycle
  • ADR-010: Production Claims, Badges, and Public Proof Source of Truth
  • ADR-013: External Target Adapters and Vector Backend Boundary