Skip to content

Pinecone Reference Backend Indexer

This page documents the Pinecone reference backend path for RAGhelm's release-control evidence workflow. Pinecone is the first reference backend, not the universal product boundary. See ADR-004 for embedding dimensions, ADR-005 for Pinecone reference backend namespace strategy, and ADR-013 for external target and backend boundaries.

Design goals

  • Provide a realistic production-grade reference vector backend.
  • Use dimension-specific Pinecone indexes when embedding dimensions differ.
  • Use namespaces inside compatible indexes to isolate retrieval strategies.
  • Use deterministic vector IDs and auditable metadata.
  • Preserve dry-run and local modes for safe development and CI.
  • Record backend/index/namespace/model/dimension evidence for traces, RAGRunManifests, and ReadinessScorecards.

Reference topology

ADR-004 rejects dimension projection as the default strategy. The reference topology uses native dimensions:

Index Intended dimension Typical use
raghelm-1536 1536 Hosted OpenAI embedding paths and production eval defaults
raghelm-1024 1024 Local/open embedding paths such as BGE large

ADR-005 owns namespace strategy inside each compatible index.

Namespace naming

{strategy}-{chunk-size-or-shape}-{model-tier}

Examples:

  • semantic-512-small
  • chunk-1024-large
  • legal-256-small

Namespaces are retrieval-strategy and access/evidence boundaries. They are not a substitute for embedding dimension compatibility.

Core API

from raghelm.ingestion.indexer import NamespaceConfig, PineconeIndexer

indexer = PineconeIndexer()
cfg = NamespaceConfig(strategy="semantic", chunk_size=512, model_tier="small")
metadata = indexer.ingestion_metadata(cfg)
indexer.upsert(cfg, records)

ingestion_metadata() emits shared provenance submodels (EmbeddingSpec, BackendSpec, and ChunkingSpec) so ingestion batches, RAGRunManifests, and retrieval experiment artifacts all use the same backend/model/index/namespace evidence. Upserts validate that every vector matches the configured index dimension before either dry-run or hosted writes proceed.

Namespace strategy definitions

RAGhelm keeps namespace names human-readable while making strategy meaning machine-readable. The MVP definitions live in NAMESPACE_STRATEGY_DEFINITIONS:

Strategy Chunking strategy Intended use
semantic recursive-semantic Reference retrieval quality experiments and production eval defaults
chunk fixed-size Fixed-size baseline comparisons
legal domain-recursive Domain-scoped/private corpus slices
test test Unit-test and dry-run fixtures only

Custom strategy names are allowed for local/private experiments but are recorded as custom chunking metadata and should not be promoted as production evidence without documentation.

Backend modes

Mode Trigger Requires Writes vectors? Intended use
cloud PINECONE_API_KEY set or explicit cloud mode Hosted Pinecone API key Yes Production/staging reference runtime
local RAGHELM_PINECONE_MODE=local Local endpoint URL Yes Development/CI with Pinecone-compatible service
auto Default convenience mode API key or local URL Yes, if configured Local convenience
dry-run --dry-run flag or dry_run=True None No Safe simulation

Mode resolution order:

  1. --dry-run / dry_run=True -> dry-run
  2. RAGHELM_PINECONE_MODE -> explicit mode
  3. PINECONE_API_KEY -> cloud
  4. PINECONE_LOCAL_URL -> local
  5. fallback -> auto, which still requires a configured backend before writes

Compatibility profile evidence

The Pinecone reference backend should prove:

  • dimension-aware index selection
  • namespace strategy isolation
  • deterministic vector IDs
  • metadata schema consistency
  • index stats and namespace counts
  • safe dry-run behavior
  • production fail-closed behavior when credentials/config are missing
  • eval results by namespace and strategy
  • retrieval policy fields recorded in manifests and scorecards
  • smoke tests that do not fake production quality claims

Testing

uv run pytest tests/test_indexer.py tests/test_pinecone_modes.py -q

Integration or smoke tests that call hosted Pinecone require explicit credentials. Unit tests should use fakes/mocks and must not make hosted calls by default.