Skip to content

ADR-005: Pinecone Reference Backend Namespace Strategy

  • Decision status: Accepted
  • Implementation status: Partial (metadata, namespace definitions, and dimension checks added)
  • Status: Accepted, amended by ADR-004 and ADR-013
  • Scope: RAGhelm Pinecone reference backend, not the universal product boundary
  • Amended by: ADR-004, ADR-013
  • Not product boundary: true
  • Applies to: RAGhelm Pinecone reference backend
  • Date: 2026-06-13
  • Amended: 2026-06-16
  • Deciders: Grey Newell
  • Tags: pinecone, reference-backend, namespaces, rag-pipeline, evidence

Amendment Note

This ADR defines the Pinecone reference backend strategy. It does not define RAGhelm’s universal product boundary.

ADR-004 amends the original “single index” assumption by requiring native dimension-specific indexes for incompatible embedding dimensions. ADR-013 further scopes Pinecone as the first reference backend while prioritizing external target adapters for the release-control product.

Corrected rule:

  • Use Pinecone as the first-class reference backend.
  • Use dimension-specific Pinecone indexes, such as raghelm-1536 and raghelm-1024, when embedding dimensions differ.
  • Use namespaces within each compatible index to isolate retrieval strategies.
  • Record index, namespace, embedding model, dimension, and backend metadata in traces, RAGRunManifest, and ReadinessScorecard evidence.

Context & Goals

RAGhelm needs a realistic production-grade reference vector backend so the reference runtime can produce credible release-readiness evidence. Pinecone is the first reference backend because it is common in production RAG, supports namespaces and metadata filtering, and gives RAGhelm a concrete compatibility profile for ingestion, retrieval, evals, traces, manifests, and scorecards.

This ADR exists to define how the RAGhelm-owned Pinecone reference runtime should organize indexes, namespaces, metadata, and evidence. It should not be read as a product-level lock-in decision. The release-control product evaluates readiness through normalized evidence, including external systems that may not use Pinecone.

We need a Pinecone reference backend namespace strategy that enables:

  • multiple retrieval strategies without quality collisions
  • safe, incremental updates with low operational overhead
  • dimension-aware embedding/index operation
  • future query routing and retrieval policy decisions
  • strategy isolation when bad data, chunking, or schema changes occur
  • namespace-level observability and regression detection
  • release-critical evidence in traces, RAGRunManifest, and ReadinessScorecard artifacts

Decision

Use Pinecone as RAGhelm’s first-class reference vector backend.

For Pinecone-backed reference runtime deployments:

  1. Use native dimension-specific indexes when embedding dimensions differ.
  2. Use namespaces inside each compatible index to isolate retrieval strategies.
  3. Treat namespace strategy, embedding model, dimension, index, and metadata schema as release-critical evidence.
  4. Do not treat Pinecone as the only backend RAGhelm can evaluate or gate.

Namespace naming convention:

{strategy}-{chunk-size-or-shape}-{model-tier}

Examples:

semantic-512-small
chunk-1024-large
legal-256-small

All vector upserts are idempotent using deterministic IDs derived from source identity, content hash, chunk strategy, and embedding model metadata. Destructive namespace-level deletes are avoided during normal operation.

The reference runtime must record these fields in traces, manifests, and scorecard evidence where applicable:

  • backend profile, for example PineconeReferenceProfile
  • backend mode: dry-run, local, staging, production
  • index name
  • namespace
  • embedding model
  • embedding dimension
  • retrieval strategy and policy version
  • metadata schema version
  • vector ID strategy
  • index stats and namespace counts
  • failure mode when Pinecone credentials or configuration are missing

Tradeoffs & Rationale

Tradeoff Option Chosen Why Rejected Alternative Reason Rejected
Backend posture Pinecone-first reference runtime; backend-neutral release evidence layer Proves RAGhelm on a real production-grade RAG stack without making Pinecone the product boundary Reference-backend-only product Too narrow; excludes external systems RAGhelm should evaluate through adapters
Index architecture Dimension-specific indexes with strategy namespaces Preserves native embedding dimensions while retaining namespace isolation One universal index Conflicts with ADR-004 because Pinecone indexes require fixed dimensions
Namespace taxonomy {strategy}-{size-or-shape}-{model-tier} Human-readable and machine-actionable for retrieval policies and eval slices Per-source or flat namespaces Causes namespace explosion and prevents strategy-based routing
Upsert model Idempotent deterministic IDs Prevents duplicate data and enables safe re-runs without manual cleanup Timestamp-based or non-deterministic IDs High risk of data corruption and manual intervention
Evidence model Record backend/index/namespace/model/dimension in release evidence Makes release gates, evals, and observability explainable Hidden backend details Makes scorecards and manifests less auditable
Product boundary External target adapters remain valid for non-Pinecone systems Keeps RAGhelm focused on release readiness rather than vector database replacement Require migration to RAGhelm reference runtime Blocks evaluation of existing customer systems

Pinecone Compatibility Profile

The Pinecone reference backend should document and test a compatibility profile that proves:

  • dimension-aware index selection, including raghelm-1536 and raghelm-1024
  • namespace strategy isolation
  • deterministic vector IDs
  • metadata schema consistency
  • index stats and namespace counts
  • safe dry-run mode
  • production fail-closed behavior when Pinecone credentials or config are missing
  • eval results by namespace and strategy
  • retrieval policy fields recorded in manifests and scorecards
  • a Pinecone smoke test that does not fake production quality claims

This profile is implementation evidence for the reference runtime. It is not a requirement that external targets use Pinecone.

Consequences

Positive

  • Gives RAGhelm a concrete production-grade reference backend for ingestion, retrieval, evals, and release evidence.
  • Failures are contained to specific dimensions, indexes, namespaces, and retrieval strategies.
  • Re-ingestion and experimentation can happen safely.
  • Namespace-level metrics can feed retrieval evals, scorecards, and manifests.
  • Aligns with ADR-004 by preserving native embedding dimensions.
  • Aligns with ADR-013 by keeping Pinecone as a reference backend while external target adapters handle other systems.
  • Supports future retrieval policies from ADR-011.

Negative / Accepted Risks

  • There are multiple indexes to provision and monitor.
  • Query/retrieval policy must choose provider, model, dimension, index, and namespace coherently.
  • Namespace count still requires active management.
  • Pinecone-first implementation may be misread as Pinecone lock-in unless docs and evidence contracts stay explicit.

Neutral

  • The architecture accepts some operational discipline in exchange for quality, debuggability, and release-control evidence.
  • Vector-store portability remains follow-on infrastructure, not the immediate product wedge.

Non-goals

  • This ADR does not make Pinecone the permanent product boundary.
  • This ADR does not prevent external target adapters from evaluating non-Pinecone systems.
  • This ADR does not require customers to migrate existing RAG systems to the RAGhelm reference runtime.
  • This ADR does not make vector-store portability the immediate product wedge.
  • This ADR does not define the universal evidence contract for all external systems; ADR-002, ADR-009, and ADR-013 own that broader release-control boundary.

Alignment with Production Requirements

This decision supports:

  • Pinecone-first reference runtime compatibility
  • retrieval strategy isolation
  • dimension-safe embedding/index operation
  • reliable re-ingestion with minimal manual intervention
  • evaluation-driven iteration and regression detection
  • release-control provenance through RAGRunManifest fields
  • backend-neutral product positioning through normalized evidence contracts and target adapters

References

  • ADR-004: Embedding Model Strategy
  • ADR-002: Release Readiness Scorecard and RAGRunManifest as Core Product Objects
  • ADR-009: RAGRunManifest Schema and Artifact Lifecycle
  • ADR-011: Retrieval Routing, Hybrid Search, and Reranking Policy
  • ADR-013: External Target Adapters and Vector Backend Boundary