Skip to content

RAGOps Observability Surface

RAGhelm's first observability slice is a local/private release-control evidence store, not a generic trace dashboard.

What is persisted

raghelm.observability.SQLiteObservabilityStore implements the shared ObservabilityStore protocol from raghelm/contracts.py and persists three contract objects in SQLite:

  • TraceRecord — privacy-preserving query/run trace metadata.
  • ReadinessScorecard — the human release decision surface.
  • RAGRunManifest — provenance for what produced the scorecard evidence.

The SQLite database stores the canonical contract JSON plus indexed fields needed by dashboards and CI:

  • run, trace, scorecard, and manifest IDs
  • target name and release candidate
  • created timestamp
  • gate status and execution mode

Privacy defaults

The store is intentionally narrow. It does not store prompt text, answer text, full retrieved context, or customer source documents by default. Trace records should use:

  • query_hash instead of raw query text
  • retrieved_chunk_ids instead of chunk content
  • PromptRef.sha256 / PromptRef.version instead of prompt bodies
  • PrivacySummary on manifests to declare whether public claims are safe

This preserves local/private debugging value while avoiding accidental prompt or context leakage.

Dashboard / metrics JSON

The Python API exposes the latest scorecard evidence at:

GET /metrics/ragops/latest?target_name=<target>&release_candidate=<optional-rc>

Configure the store path with:

export RAGHELM_OBSERVABILITY_DB=.raghelm/observability.sqlite3

If no store is configured, or no scorecard exists, the endpoint fails closed with 404 instead of inventing evidence.

The response is derived from persisted scorecard + manifest contracts and includes:

  • latest gate decision and policy version
  • retrieval / generation / agent quality metrics
  • cost and latency summaries
  • pass/review/block check counts and check evidence
  • regression deltas against the previous scorecard for the target, where available
  • manifest source-control, corpus, backend, prompt, judge, artifact hashes, and privacy summary
  • badge source-of-truth eligibility

Production/public badges are only eligible when the linked manifest is production mode, safe_for_public_claims is true, and the gate decision is ship.

Minimal usage

from raghelm.contracts import ReadinessScorecard, RAGRunManifest, TraceRecord
from raghelm.observability import SQLiteObservabilityStore, latest_ragops_summary

store = SQLiteObservabilityStore(".raghelm/observability.sqlite3")
store.record_manifest(manifest)      # RAGRunManifest
store.record_scorecard(scorecard)    # ReadinessScorecard
store.record_trace(trace)            # TraceRecord

summary = latest_ragops_summary(store, target_name="reference-rag")

Non-goals for this slice

  • Replacing Langfuse, Phoenix, LangSmith, OpenTelemetry, or OpenInference.
  • Storing raw prompt/context/customer content.
  • Building a generic observability dashboard.
  • Treating raw eval summaries as production badge proof without scorecard + manifest linkage.

Related issues: #63, #69, #82, #92, #93.