Skip to content

ADR-001: Product Architecture Spine for Private RAGOps Release Control

  • Decision status: Accepted
  • Implementation status: Partial
  • Applies to: Reference runtime, external targets, enterprise/BYOC
  • Date: 2026-06-16
  • Deciders: Grey Newell
  • Tags: product-architecture, ragops, release-control, agent-readiness, investability

Context

RAGhelm has several overlapping product and architecture directions:

  • broad production RAG platform capabilities: ingestion, retrieval, generation, routing, hybrid search, reranking, observability, deployment
  • private-compute RAGOps capabilities: readiness scorecards, release gates, provenance artifacts, production quality claims, trace interoperability, BYOC deployment
  • business-agent readiness capabilities: typed tools, state, budgets, memory, safety, human review, business outcome instrumentation
  • adoption/demo assets: positioning narrative, user profile, proof artifacts, pilot story, and demo script

Trying to make all of these the primary product at once creates an unfocused architecture. A broad RAG platform would compete directly with application frameworks and RAG platforms while exposing unfinished implementation gaps. A domain-specific agent product would be concrete but too narrow as the primary infrastructure wedge.

The project needs one product architecture spine that determines which capabilities are core, which are reference/demo work, and which are future extensions.

Decision

RAGhelm's primary product architecture spine is private RAGOps and release-control infrastructure, demonstrated through a synthetic business-agent reference workflow.

In practical terms:

  • RAGhelm's core product object is a release-readiness decision for private RAG and agent systems.
  • The built-in RAG pipeline and synthetic business-agent scenario are reference targets used to prove and demo the release-control workflow.
  • RAGhelm should not position itself as a general-purpose replacement for RAG application frameworks, agent frameworks, or observability platforms.
  • The product should answer: "Can this private RAG or agent release ship, and why or why not?"
  • Future integrations should let RAGhelm evaluate and gate external RAG/agent stacks rather than requiring every team to adopt RAGhelm's internal RAG implementation.

This creates a hybrid architecture:

Private RAG / agent system under test
  -> RAGhelm evaluation and trace capture
  -> provenance artifact
  -> readiness scorecard
  -> release gate: ship / block / needs review
  -> dashboard, CI/CD, and public proof and stakeholder-demo artifacts

Reference target:
  synthetic business-agent workflow

Consequences

Positive

  • Creates a sharper product wedge than a broad RAG platform.
  • Keeps initial implementation focused on release readiness, scorecards, provenance, and demo clarity.
  • Supports business-agent demos by showing production-agent judgment, not only chatbot construction.
  • Makes evals, observability, cost, latency, provenance, and gates first-class product concerns.
  • Leaves room for external target adapters so RAGhelm can evaluate RAGFlow, LlamaIndex, LangChain, custom APIs, or internal agent systems later.
  • Reduces pressure to immediately implement every promised RAG feature as a core platform capability.

Negative

  • Requires disciplined public positioning so users do not expect a complete RAG application platform.
  • Some README claims and older milestone issues may need to be downgraded, reframed, or moved behind the release-control story.
  • The built-in RAG/agent implementation becomes a reference target, which may disappoint users looking for an all-in-one RAG app framework.
  • The product must define a strong scorecard/release-gate object quickly; otherwise the wedge remains abstract.

Neutral / Trade-offs

  • Pinecone, FastAPI, retrieval strategies, model routing, and agent tooling remain important, but they serve the release-readiness workflow rather than defining the product by themselves.
  • Synthetic business-agent workflows remain useful for demos, but they should not narrow the product architecture.
  • External interoperability becomes strategically important, but it can follow the initial reference implementation once the scorecard and gate semantics are stable.

Alternatives Considered

  1. Broad production RAG platform

Pros: - Impressive full-stack demo if complete. - Directly showcases ingestion, retrieval, generation, routing, evals, observability, and deployment.

Cons: - Too broad for the focused release-control wedge. - Competes directly with established RAG frameworks and platforms. - Existing issues already show implementation gaps around hybrid retrieval, reranking, router behavior, agent runtime, deployment, and docs alignment.

  1. Private-compute RAGOps / release-control only

Pros: - Clear infrastructure wedge. - Strong fit for scorecards, release gates, provenance, CI/CD, trace import/export, and BYOC.

Cons: - Needs a concrete reference workload to make the demo emotionally and commercially legible. - Can feel abstract without a business-agent scenario.

  1. Customer-specific production-agent readiness platform only

Pros: - Strong role-alignment story. - Naturally highlights typed tools, safety, memory, escalation, and business outcomes.

Cons: - Too narrow as the primary product architecture. - Pulls too much agent runtime complexity into the first build. - Risks becoming a consulting/demo artifact rather than an investable platform wedge.

  1. Hybrid: private RAGOps spine plus business-agent reference workflow

Pros: - Combines an infrastructure wedge with a concrete reference workflow. - Makes release readiness the product while using a business-agent scenario to prove value. - Keeps the internal RAG/agent stack as a reference target, not a promise to replace every framework.

Cons: - Requires careful documentation to keep the reference implementation distinct from the product boundary.

This is the chosen option.

Implementation Guidance

Initial implementation should prioritize:

  • defining the readiness scorecard and release-gate semantics
  • making production quality claims fail closed when backed artifacts are missing or stale
  • producing a clear synthetic reference workflow
  • showing provenance for eval/release runs
  • aligning README and docs around release control rather than broad RAG-platform claims

Work that should remain secondary until the release-control spine is clear:

  • full general-purpose agent runtime
  • full vector backend abstraction
  • broad provider plugin ecosystem
  • hybrid retrieval and reranking as default behavior
  • external target adapters beyond the first proof path
  • polished commercial packaging and web presence

References

  • ADR-005: Pinecone Reference Backend Namespace Strategy
  • ADR-004: Embedding Model Strategy