Evaluation Overview¶
Evaluation is a core input to RAGhelm’s release-control layer. Evals measure retrieval, generation, regression, latency, and production-readiness evidence, but eval summaries alone are not the product object. Per ADR-002, release decisions should be expressed through ReadinessScorecards linked to RAGRunManifests.
What the eval suite provides¶
- Golden dataset loading and validation
- Retrieval metrics: Recall@k, MRR, NDCG@k
- Generation scoring: faithfulness, relevance, completeness, overall
- Regression checks against baselines
- Production-mode hooks for Pinecone retrieval and LLM judging
- Machine-readable JSON artifacts under
data/eval_results/
Local vs production evals¶
| Mode | Purpose | Evidence status |
|---|---|---|
local |
Fast deterministic development feedback | Useful for development; not production proof |
production |
Hosted Pinecone retrieval plus production judge behavior | Eligible as input to scorecards/manifests when config and artifacts are valid |
Public claims and badges must fail closed unless backed by production scorecard and manifest evidence per ADR-010.
Key components¶
| Component | Purpose |
|---|---|
golden_dataset.py |
Load and validate evaluation datasets |
runner.py |
Orchestrate eval runs |
metrics.py |
Compute Recall@k, MRR, NDCG@k |
scorer.py |
Score generation quality |
regression.py |
Detect regression against baseline |
benchmark.py |
Latency benchmarking helpers |
production_adapter.py |
Production-mode retrieval/generation/judge integration |
Golden dataset sources¶
The default 100-example dataset uses open-license RPG/system-reference content. It is a development and demonstration dataset, not customer production data.
| Source | License | Examples |
|---|---|---|
| Cairn RPG | CC-BY-SA 4.0 | 20 |
| SCP Foundation | CC-BY-SA 3.0 | 20 |
| Fate Core / related SRDs | CC-BY 3.0 | 21 |
| Dungeon World | CC-BY 3.0 | 20 |
| D&D 5.1 SRD | CC-BY 4.0 | 20 |