Skip to content

ADR-019: Async Execution and Worker Queue

  • Decision status: Accepted
  • Implementation status: Not started
  • Applies to: Reference runtime, future
  • Date: 2026-06-16
  • Deciders: Grey Newell
  • Tags: async, workers, queues, evals, agents, ci-cd, release-control

Context

RAGhelm needs to run work that may exceed a synchronous HTTP request lifecycle: production evals, release checks, benchmark runs, external target runs, and reference agent workflows. ADR-007 makes the shared QueryPipeline the source of truth for served and evaluated behavior. ADR-015 defines SQLite for local persistence and a Postgres-compatible production path. ADR-016 scopes RAGhelm toward agent readiness evaluation with a limited reference agent.

The execution architecture must support local/demo usage without heavy infrastructure while leaving a path toward production worker backends and CI release workflows.

This decision supports eval/admin execution, deployable async workers, and benchmark/proof artifacts without making issue numbers part of the durable architecture contract.

Decision

Use a staged async execution model:

  1. Now: background worker with a simple local queue.
  2. Later production option: Redis/RQ/Celery/Temporal-style worker backend.
  3. Later CI/release option: GitHub Actions or CI workflows for long-running release checks and public proof artifacts.

The first implementation should not require Redis, Celery, Temporal, or another external queue. Local/demo/open-source mode should use an in-process or lightweight background worker with job state persisted through the ADR-015 SQLite store.

Production/BYOC deployments may later swap the local queue for a stronger worker backend. CI workflows may later run release checks and emit scorecard/manifest bundles for published proof and badges.

Consequences

Positive

  • Keeps local development and demos simple.
  • Supports dashboard polling, cancellation, retries, and persisted job state without immediate heavy infrastructure.
  • Leaves a clear production path for scalable/reliable workers.
  • Leaves a clear CI path for public proof artifacts and published badges.
  • Aligns async runs with scorecards, manifests, traces, and persistence.

Negative

  • The first local queue will not provide production-grade durability or horizontal scaling.
  • Production deployments will need a later worker-backend milestone.
  • CI/release workflows will need a separate artifact publication path.

Neutral / Trade-offs

  • Synchronous CLI execution can remain available for simple local workflows.
  • Local background jobs should be treated as convenience, not production reliability proof.
  • CI release checks complement but do not replace product/dashboard async execution.

Alternatives Considered

  1. Synchronous CLI/API only

Pros: - Simplest implementation.

Cons: - Poor UX for long evals and agent runs. - Weak support for retries, cancellation, dashboard status, and production operations.

  1. Background worker with simple local queue

Pros: - Good local/demo next step. - Supports async jobs without external infrastructure. - Can persist state using SQLite.

Cons: - Not enough for high-scale or highly reliable production use.

This is the chosen immediate option.

  1. Redis/RQ/Celery/Temporal-style worker system

Pros: - Stronger production execution model. - Better reliability, retries, concurrency, and operations.

Cons: - Adds deployment and dependency complexity too early.

This is a later roadmap item.

  1. GitHub Actions/CI only for long-running jobs

Pros: - Strong for release gates and public artifacts. - Fits published badge/proof workflows from ADR-010.

Cons: - Not enough for local dashboard workflows, product APIs, or interactive target evaluation.

This is a later roadmap item for release/public proof workflows.

Job Record Shape

Async jobs should persist:

  • job_id
  • run_id
  • target_id
  • job_type: eval, release_check, benchmark, agent_run, target_run
  • status: queued, running, succeeded, failed, cancelled, expired
  • created_at, started_at, completed_at
  • attempt_count
  • max_attempts
  • requested_by when available
  • scorecard_id when produced
  • manifest_id when produced
  • trace_ids
  • artifact_paths
  • failure_mode
  • error_summary
  • cancellation reason when cancelled

API and Worker Guidance

Initial endpoints may include:

POST /runs
GET /runs/{run_id}
GET /runs/{run_id}/trace
POST /runs/{run_id}/cancel

For agent-specific APIs, use the same underlying job model rather than a separate queue.

The local worker should:

  • persist job state before execution starts
  • mark terminal states reliably
  • write scorecard/manifest artifacts on success
  • write failure artifacts or structured failure modes on failure
  • support cancellation when safe
  • avoid duplicate side effects through idempotency keys where tools mutate state

Roadmap Items Required

Create follow-on roadmap/issues for:

  1. production worker backend support using Redis/RQ/Celery/Temporal-style infrastructure
  2. CI/GitHub Actions release-check runner that emits scorecard + RAGRunManifest artifact bundles for published proof and badges

Implementation Guidance

Initial implementation should:

  • define job schema and persistence interface
  • implement local SQLite-backed job queue
  • connect eval/release-check/agent target runs to the queue
  • add status/result/trace endpoints
  • make dashboard poll job status
  • preserve scorecard/manifest/run linkage

Non-goals:

  • production-grade distributed queue in the first implementation
  • CI-only execution as the only async path
  • irreversible tool actions without idempotency and human approval
  • hiding failed jobs without structured error artifacts

References

  • ADR-007: Shared QueryPipeline as Production-Eval Source of Truth
  • ADR-010: Production Claims, Badges, and Public Proof Source of Truth
  • ADR-015: Persistence Store for Release-Control History
  • ADR-016: Agent Runtime Scope, Tool Contract, and Side-Effect Safety