Skip to content

ADR-012: Model Provider Abstraction Boundary

  • Decision status: Accepted
  • Implementation status: Not started
  • Applies to: Reference runtime
  • Date: 2026-06-16
  • Deciders: Grey Newell
  • Tags: model-providers, embeddings, generation, judging, cost, observability

Context

RAGhelm uses model calls in several release-critical layers:

  • embedding source documents and queries
  • grounded generation
  • LLM-as-judge or offline judge scoring
  • optional routing or classification
  • future cost optimization and private/local model support

ADR-004 defines the embedding dimension strategy and rejects projection. ADR-002 and ADR-009 require scorecards and RAGRunManifests to record model/provider choices as provenance. ADR-007 requires served query and production eval behavior to share one QueryPipeline.

This decision supports related implementation work without making issue numbers part of the durable architecture contract.

Decision

Use one small model provider registry covering:

  1. embeddings
  2. generation
  3. judges/scorers

The provider abstraction should not be a broad plugin system at first. It should define small, typed interfaces for the model calls that affect release-control evidence.

The registry must expose enough metadata for scorecards, manifests, traces, and cost accounting:

  • provider name
  • model name
  • model role: embedding, generation, judge
  • embedding dimension where applicable
  • context/token limits where applicable
  • pricing/cost metadata when known
  • locality/privacy class: hosted, private-cloud, local
  • supported modes: local, demo, staging, production
  • failure semantics and retry policy

Routing/classification model calls may use the same registry when they become model-backed, but the first abstraction boundary is embeddings + generation + judges.

Consequences

Positive

  • Gives scorecards and RAGRunManifests consistent model provenance.
  • Makes provider swaps explicit and testable.
  • Supports private/local model positioning without duplicating ad hoc wrappers.
  • Centralizes cost, latency, and failure metadata for model calls.
  • Keeps provider abstraction focused enough for initial implementation.

Negative

  • Requires refactoring any direct provider calls into registry-backed adapters.
  • Adds interface design work before every provider is fully supported.
  • Requires tests for provider failures, missing credentials, and production fail-closed behavior.

Neutral / Trade-offs

  • This is not a general plugin system for every tool or runtime.
  • Providers can be implemented incrementally behind the registry.
  • Local/demo providers may return deterministic outputs, but production release evidence must identify them as local/demo and must not masquerade as hosted production output.

Alternatives Considered

  1. Embeddings only

Pros: - Directly addresses dimension/index issues. - Smaller initial interface.

Cons: - Too narrow for release-control provenance because generation and judges also determine quality, cost, latency, and trust.

  1. Generation only

Pros: - Helps replace generator stubs with provider-backed grounded generation.

Cons: - Does not address embedding dimensions, retrieval compatibility, or judge provenance.

  1. Embeddings + generation + judges through one provider registry

Pros: - Covers all model calls that materially affect scorecards and manifests. - Keeps provenance, cost accounting, and failure handling consistent. - Fits ADR-004, ADR-002, ADR-007, and ADR-009.

Cons: - Requires a slightly broader interface than an embeddings-only wrapper.

This is the chosen option.

  1. Full plugin system for every provider/tool/runtime

Pros: - Maximum extensibility.

Cons: - Too much surface area too early. - Harder to audit, test, and explain.

Interface Guidance

Minimum embedding provider contract:

  • embed_documents(texts, *, model, mode, trace_context) -> vectors + usage metadata
  • embed_query(text, *, model, mode, trace_context) -> vector + usage metadata
  • exposes dimension, provider, model, token/cost estimates, and failure mode

Minimum generation provider contract:

  • generate_grounded_answer(query, context, *, model, prompt_version, mode, trace_context) -> answer + citations + usage metadata
  • exposes provider, model, token/cost estimates, latency, finish reason, and failure mode

Minimum judge provider contract:

  • score_answer(example, answer, context, *, rubric_version, model, mode, trace_context) -> structured scores + usage metadata
  • supports offline heuristic and production LLM judge modes explicitly

Production Safety Rules

  • Production mode must fail closed when required credentials or provider configuration are missing.
  • Local/demo providers must be labeled in traces, manifests, scorecards, and public proof artifacts.
  • Provider fallback must be explicit and recorded; no silent fallback from production provider to local/mock provider.
  • Embedding dimensions must be validated against selected index dimensions.
  • Cost and latency metadata should be captured for every provider call when available.
  • Model/provider changes require eval comparison against baseline before promotion.

Required Observability and Manifest Fields

Every model call that affects release evidence should record:

  • provider name
  • model name
  • model role
  • mode
  • token counts where applicable
  • cost estimate where applicable
  • latency
  • retry/fallback behavior
  • failure mode
  • embedding dimension where applicable
  • prompt/rubric version where applicable

References

  • ADR-004: Embedding Model Strategy
  • ADR-002: Release Readiness Scorecard and RAGRunManifest as Core Product Objects
  • ADR-007: Shared QueryPipeline as Production-Eval Source of Truth
  • ADR-009: RAGRunManifest Schema and Artifact Lifecycle