Golden dataset
The trusted answer key for your AI tests: curated real inputs paired with the exact result or behavior the system should produce.
See it
What it is
A golden dataset is the small, curated set of inputs and expected behavior used as the stable benchmark for an AI feature. Each case may carry an exact answer, a list of facts that must appear, the right tool call, or a human-written scoring rubric. It is the answer key behind an eval, not the pile of examples used to train the model.
Start it with real production cases: common requests, costly failures, edge cases, and examples experts disagree about. Have a domain expert review every label, version the set with the prompt and model, and keep a held-out slice hidden from anyone tuning the system.
Gotcha: gold can tarnish. Policies change, products gain features, and an answer that was correct six months ago can become a regression trap. Record who approved each case and when, review stale cases, and do not keep peeking at the held-out answers until the system has memorized the benchmark instead of improving.
Ask AI for it
Create a versioned golden dataset in JSONL for this AI feature. Define a JSON Schema requiring id, category, input, expected_behavior, forbidden_behavior, scoring_method, source, reviewer, approved_at, and last_verified_at. Seed it with real happy paths, production failures, boundary cases, and adversarial cases; keep exact-match expectations separate from rubric-based ones. Produce a stratified development split and a sealed holdout split, compute a SHA-256 digest for every released version, reject duplicate ids in CI, and never place holdout examples in prompts, fine-tuning data, or synthetic-data seeds.