Implementation Workflow

RAGhelm's implementation queue is controlled by issue #99, not by the older issue order. The minimum investable product is:

CLI + dashboard + CI gate that evaluates one existing RAG/agent target, emits a readiness scorecard, blocks a bad release, and produces a provenance manifest in a private/local deployment.

This workflow keeps the project from expanding into a generic RAG platform, generic eval platform, generic observability platform, or generic agent framework.

Product boundary

Build only work that strengthens one of these paths:

1. Release-readiness proof path: canonical query/eval pipeline, scorecard, gate, manifest, persisted evidence, dashboard, CI release check.
2. ACQ Vantage role demo: narrow business-agent workflow that proves production-agent readiness without becoming a broad autonomous-agent platform.

Park work that does neither.

Non-goals for the investable MVP

• full agent-framework breadth
• generic observability-platform replacement
• full onchain proof layer
• many vector backends
• many vertical packs
• production SaaS
• generic RAGFlow-style workflow builder

Required first PR: contract freeze

Before parallel implementation begins, merge one contract-freeze PR. It must not implement the whole product. It freezes shared object shapes, boundaries, CLI/API names, and fixtures so parallel agents do not create incompatible abstractions.

The contract-freeze PR owns:

• shared contracts in raghelm/contracts.py
• golden contract fixtures in tests/fixtures/contracts/
• contract tests in tests/test_contracts.py
• documentation of CLI commands, API response shapes, and module ownership

Core objects frozen by this PR:

• QueryPipeline
• QueryRequest
• QueryResult
• TargetAnswer
• ReadinessScorecard
• ReleaseGate
• RAGRunManifest
• TraceRecord
• TargetAdapter
• ObservabilityStore
• AgentRun

The contract freeze directly covers #57, #92, #64, #66, #63, and enough of #72/#73 to prevent later rewrites.

Rationale:

• 57: /query and production eval currently use different orchestration paths and need a shared QueryPipeline.

• 92: scorecard and gate must become first-class product concepts.

• 64: evals/releases need a provenance artifact tying together git SHA, corpus, chunking, embedding, backend, routing, prompt, judge, metrics, and artifact hashes.

• 63: observability needs a stable trace/evidence shape before the SQLite/dashboard work starts.

• 72/#73: the agent demo needs typed runs, steps, budgets, tool side-effect classification, and human-approval boundaries without a broad agent framework.

Branch and PR conventions

Use one integration branch naming pattern per lane:

• feat/contracts-*
• feat/pipeline-*
• feat/ingestion-*
• feat/scorecard-*
• feat/observability-*
• feat/agent-demo-*

Every PR must include:

• tests
• docs updates when behavior, commands, contracts, or architecture posture change
• artifact examples when the work emits scorecards, manifests, traces, eval summaries, or CI bundles
• a note on claim honesty: local/demo output must not be presented as production evidence

Parallel workstreams after contracts merge

Workstream A: canonical query/eval pipeline

Owns #55, #56, #57, #58, #59, and useful parts of #60.

Primary modules:

• raghelm/query/ or raghelm/pipeline.py
• raghelm/generation/
• raghelm/api/
• eval adapter code in raghelm/eval/

Responsibilities:

• make Python FastAPI the documented canonical query path
• replace implied generator stub behavior with provider-backed generation while keeping explicit stub mode
• route /query and eval through the same QueryPipeline
• emit routing metadata
• fail closed for production badges and production claims

Workstream B: ingestion, backend, and retrieval experiments

Owns #68, #61, #62, #84, and older #6/#7/#8/#10/#11/#24/#27 only as implementation subtasks.

Primary modules:

• raghelm/ingestion/
• raghelm/vector/
• retrieval experiment utilities

Responsibilities:

• unify ingestion/indexing paths
• make ingestion reproducible
• make namespace strategy definitions explicit
• keep vector dimensions consistent
• make retrieval experiment results comparable
• add vector backend abstraction interface-first; implement a minimal self-hosted adapter only if required for MVP

Workstream C: release-readiness product object

Owns #92, #93, #101, #59, #64, and parts of #94.

Primary modules:

• raghelm/readiness/
• raghelm/provenance/
• benchmarks/
• CI release-check scripts

Responsibilities:

• emit a scorecard artifact
• evaluate a deterministic gate policy
• emit a RAGRunManifest
• produce a CI release-check bundle
• create a benchmark where a bad candidate fails and an improved candidate passes
• validate artifact hashes

Workstream D: local/private observability and dashboard

Owns #63, #69, #82, and dashboard parts of #92/#93.

Primary modules:

• raghelm/observability/
• SQLite schema/migrations
• dashboard scorecard views
• metrics endpoints

Responsibilities:

• persist traces, cost/latency metrics, eval summaries, regression diffs, and badge/source-of-truth summaries
• avoid sensitive prompt/context storage by default
• show a RAGOps scorecard surface, not a generic query UI
• display latest gate decision, regression deltas, retrieval/generation/agent quality, cost/latency, manifest hashes, and production/local status

Workstream E: role-aligned agent demo

Owns #71, #72, #73, and only the minimum of #74/#75/#76/#77/#79/#81/#83 needed for a credible demo.

Primary modules:

• raghelm/agents/
• raghelm/tools/
• raghelm/memory/ only if essential
• synthetic demo data

Responsibilities:

• implement a narrow ACQ Vantage-style synthetic workflow
• use a small typed state-machine runtime, not a general autonomous-agent platform
• include typed runs, steps, state, budgets, trace IDs, deterministic replay, stop reasons, and bounded-loop constraints
• include typed tool specs, validation, side-effect classification, auth scope, retry policy, and human approval for high-risk side effects

Contract tests as the integration gate

Before merging any feature PR, it must pass contract tests proving it emits or consumes the shared objects correctly:

• QueryResult
• ReadinessScorecard
• RAGRunManifest
• TraceRecord
• TargetAnswer
• AgentRun

Contract tests are the guardrail that allows maximum parallelism without incompatible local designs.

Quality gates for every PR

1. Local deterministic gate

Run unit tests, offline evals, schema validation, and artifact snapshot tests. This gate must not require hosted credentials.

2. Production-capable smoke gate

Verify that missing credentials fail closed and configured credentials call real providers. Do not silently downgrade production behavior into local/mock behavior.

3. Claim-honesty gate

Verify that README/docs/badges do not claim production metrics unless backed by a valid production summary or CI artifact bundle. Local/demo results must be labeled as local/demo evidence.

Integration merge order

1. contracts
2. query pipeline
3. scorecard/manifest
4. CI gate
5. observability store
6. dashboard
7. target adapter
8. agent demo
9. benchmark proof
10. investor/business docs

Batch priority

Batch 1: coherent investable product

55 + #56 + #57 + #58 + #68 + #92 + #64 + #63 + #93 + #101

Outcome: canonical API, explicit real/stub generation mode, shared pipeline, strategy routing, reproducible ingestion metadata, scorecard, manifest, persisted evidence, benchmark proof, and CI release gate.

Batch 2: credible company/investor artifact

66 + #89 + #90 + #87 + #88 + #94 + #95 + #96 + #97

Outcome: external target adapter, private-cloud story, security pack, competitive matrix, packaging/pricing, ROI model, pilot package, investor demo, and moat narrative.

Batch 3: role-alignment layer

71 + #72 + #73 + #74 + #75 + #79 + #81 + #83

Outcome: production-agent demo, internal-system tool use, golden workflow evals, cost controls, observability, TypeScript/Python boundary, and measurable business outcomes.