Golden Dataset¶
The golden dataset is the default development dataset used to evaluate retrieval and generation quality. It is useful for local development, regression testing, and reference-runtime demonstrations. Production release gates should use project-specific datasets and scorecard/manifest evidence.
Structure¶
Each example follows this schema:
{
"id": "cairn_001",
"question": "What happens when a character reaches 0 STR in Cairn?",
"expected_answer": "When a character reaches 0 STR, they are DEAD.",
"source_docs": ["cairn_srd.md"],
"relevant_chunks": ["When a character reaches 0 STR, they are DEAD."],
"difficulty": 1,
"category": "factual_lookup"
}
Fields¶
| Field | Type | Description |
|---|---|---|
id |
string | Stable unique identifier. |
question |
string | Query to evaluate. |
expected_answer |
string | Ground-truth answer. |
source_docs |
list[string] | Source document filenames. |
relevant_chunks |
list[string] | Reference chunks containing the answer. |
difficulty |
int | 1 = easy, 2 = medium, 3 = hard. |
category |
string | Query type. |
Query categories¶
Valid categories:
factual_lookupcomparisonsynthesistemporalcontradictory
These categories let scorecards and retrieval policies report quality by query class rather than only aggregate averages.
Sources¶
All 100 default examples use open-license RPG/system-reference content:
| Source | License | Count |
|---|---|---|
| Cairn RPG SRD | CC-BY-SA 4.0 | 20 |
| SCP Foundation Wiki | CC-BY-SA 3.0 | 20 |
| Fate Core / related SRDs | CC-BY 3.0 | 21 |
| Dungeon World SRD | CC-BY 3.0 | 20 |
| D&D 5.1 SRD | CC-BY 4.0 | 20 |
Validation¶
The dataset is validated on every eval run using raghelm.eval.golden_dataset.validate_dataset():
from raghelm.eval.golden_dataset import load_golden_dataset, validate_dataset
dataset = load_golden_dataset("raghelm/eval/golden_dataset.json")
issues = validate_dataset(dataset)
if issues:
raise SystemExit(issues)
Validation checks:
- no duplicate IDs
- required fields are present
- difficulty is 1, 2, or 3
- category is one of the valid types
Creating custom datasets¶
Custom production datasets should use stable IDs, source document references, relevant chunks, category, and difficulty. Avoid embedding private source content in public proof bundles; manifests may reference private artifacts by hash/path/URI instead.