Regression Testing

Regression testing detects quality drops between eval runs. Regression results are evidence inputs for release gates; they are not by themselves a complete release decision.

How it works

1. The first run saves data/baseline.json with current metrics.
2. Later runs compare against the baseline.
3. If any metric drops beyond tolerance, the run fails.

CLI output

When a regression is detected:

Regression detected
  recall@5: 1.0000 -> 0.8500 (-15.00%)
  mrr: 1.0000 -> 0.9200 (-8.00%)

The command exits non-zero so CI can block the release workflow.

Resetting a local baseline

rm data/baseline.json
uv run python -m raghelm eval --suite full

Do not reset production baselines casually. Production baseline changes should be linked to scorecard/manifest evidence and reviewed under the applicable release policy.