System Overview¶

RAGhelm is private RAG and agent release-control infrastructure. Its purpose is to produce auditable release evidence for private RAG/agent systems: readiness scorecards, RAGRunManifests, deterministic gate results, eval artifacts, and traces.

This page separates current implementation from target architecture so readers can tell what exists today from what the ADRs have accepted as the roadmap direction.

Product flow¶

flowchart TD
    Target[Reference runtime or external RAG/agent target] --> Evidence[Normalized run evidence]
    Evidence --> Eval[Evaluation and scoring]
    Evidence --> Manifest[RAGRunManifest provenance]
    Eval --> Scorecard[ReadinessScorecard]
    Manifest --> Scorecard
    Policy[Release policy] --> Scorecard
    Scorecard --> Gate{Release gate}
    Gate --> Ship[ship]
    Gate --> Review[needs_review]
    Gate --> Block[block]
    Scorecard --> Dashboard[Dashboard / CI / proof bundle]

Runtime boundaries¶

Boundary	Role	Current implementation status
QueryPipeline	Canonical path for RAGhelm-owned reference runtime behavior	Contract frozen in `raghelm/contracts.py`; current API/eval paths are not fully unified
TargetAdapter	Canonical boundary for black-box or external systems under test	Contract frozen in `raghelm/contracts.py`; implementation not started
RAGRunManifest	Provenance artifact for run evidence and release artifacts	Contract frozen in `raghelm/contracts.py`; partial local demo JSON artifact
ReadinessScorecard	Human/product release decision artifact	Contract frozen in `raghelm/contracts.py`; partial local demo JSON artifact
TypeScript gateway	Product/public API boundary	Target architecture; current Fastify package is lightweight
Python runtime	AI runtime for ingestion, retrieval, generation, eval, scorecards	Partially implemented; current canonical AI surface
SQLite observability store	Local/private persisted traces, scorecards, manifests, latest RAGOps evidence JSON	First vertical slice implemented in `raghelm/observability/`

Boundary rule from ADR-007 and ADR-013:

QueryPipeline is canonical for RAGhelm-owned reference runtime behavior.
TargetAdapter is canonical for black-box or external systems under test.
Both normalize evidence into RAGRunManifest and ReadinessScorecard artifacts.

Current user entry points¶

User goal	Start here	Command or URL
Install locally	Installation	`uv sync --extra dev`
Run a smoke-quality local eval	Quick Start	`uv run python -m raghelm eval --suite quick`
Run production eval against Pinecone and LLMs	Running Evaluations	`uv run python -m raghelm eval --suite full --mode production --judge-mode production`
Ingest documents safely	CLI Commands	`uv run python -m raghelm ingest ./corpus --dry-run --namespace default`
Understand Pinecone reference backend index/namespace design	Pinecone Reference Backend	Read ADR-004 and ADR-005 first
Review architecture decisions	Architecture	Check decision and implementation status
Run dashboard locally	Dashboard package	`cd packages/dashboard && pnpm install && pnpm dev`

Implemented components¶

Ingestion CLI¶

Implemented in raghelm/ingestion/__init__.py and exposed through python -m raghelm ingest.

Responsibilities:

read .md and .txt source files
chunk content with bounded overlap
compute chunk checksums
estimate token counts and embedding cost
embed with OpenAI in non-dry-run mode
upsert vectors into Pinecone namespaces
write an ingestion audit artifact to data/ingestion_audit.json

Pinecone indexer¶

Implemented in raghelm/ingestion/indexer.py.

Responsibilities:

enforce backend mode semantics: cloud, local, auto, dry-run
create or connect to dimension-specific Pinecone indexes
upsert batches with deterministic vector IDs
return namespace-level index stats

ADR-005 defines the Pinecone reference backend namespace strategy. ADR-004 owns embedding dimensions and the reference indexes raghelm-1536 and raghelm-1024. ADR-013 prevents this Pinecone reference profile from becoming the universal product boundary.

Retrieval¶

Implemented in raghelm/retrieval/retriever.py.

Responsibilities:

embed queries with BAAI/bge-large-en-v1.5
query the configured Pinecone index and namespace
return chunk IDs, scores, metadata, and text previews
provide deterministic empty results when Pinecone is not configured or the index is unavailable

Default retrieval index: raghelm-1024.

Query API¶

Implemented in raghelm/api/server.py as a Python FastAPI application.

Endpoints:

Endpoint	Purpose
`GET /health`	Service health and Pinecone configuration status
`POST /query`	Classify complexity, retrieve context, generate answer, and return cost metadata
`GET /metrics/cost`	Aggregated request cost metrics
`GET /metrics/index`	Pinecone index and namespace stats

Per ADR-006, Python remains the canonical implemented AI runtime. The TypeScript Fastify package should become the product gateway later but does not currently own AI behavior.

Contract-freeze note: downstream query/eval work should converge /query and production eval output on the shared QueryResult, TargetAnswer, and RoutingMetadata contracts in raghelm/contracts.py. The current endpoint response remains backward-compatible until the canonical QueryPipeline implementation lands.

Evaluation¶

Implemented in raghelm/eval/ and exposed through python -m raghelm eval.

Modes:

Mode	Purpose	External dependencies
`local`	Fast deterministic development eval with mocked retrieval/generation	none
`production`	Real Pinecone retrieval plus LLM generation path	Pinecone + OpenAI and/or Anthropic keys

Local evals are useful for development mechanics but must not be represented as production release evidence.

Dashboard¶

Implemented in packages/dashboard with React and Vite.

Current local demo purpose:

show a ReadinessScorecard and linked RAGRunManifest
demonstrate release-control framing
provide local visual proof of scorecard/manifest concepts

The dashboard is not yet the authoritative production scorecard store.

Architecture invariants¶

Production quality claims must come from a production ReadinessScorecard linked to a valid production RAGRunManifest.
Local/demo artifacts must be clearly labeled and must not back production badges or public quality claims.
Retrieval metrics and generation metrics remain separate.
QueryPipeline is canonical for RAGhelm-owned reference runtime behavior.
TargetAdapter is canonical for external systems under test.
Pinecone namespace/index decisions follow ADR-005 and ADR-004.
Private source content, credentials, and sensitive traces must not appear in public proof artifacts by default (ADR-003).
ADR implementation status in the architecture index is authoritative for target-vs-implemented posture.