# Golden Dataset

A curated, versioned set of representative inputs with trusted expected outputs used to score every change.

## Why the CCAR-P exam tests this

The exam checks that you build a stable reference set before optimizing, so scores are comparable over time.

## Definition

A **golden dataset** is a fixed, hand-verified collection of inputs paired with **expected outputs** or grading **rubrics**. It is the yardstick every eval run measures against, so its quality caps the quality of your entire evaluation. Items should be **representative** of real traffic and deliberately include hard cases, edge cases, and known failure modes — not just easy happy-path examples.

Treat it like code: **version** it, review changes, and keep it stable so scores are comparable across runs. When you fix a new class of bug in production, add a case for it to the set so the fix is protected going forward. Coverage should span your **objectives** and **domains** rather than clustering on whatever was easy to collect.

A golden set is the shared substrate for `eval-driven-development`, `regression-testing`, and `llm-as-judge` — all of them run over it. Keep held-out slices so you can detect overfitting to the set itself, and avoid leaking the golden answers into prompts or fine-tuning data.

## Exam trap

A golden set that only contains easy, happy-path examples inflates every score. Coverage of hard and failure cases is what makes it trustworthy.

## Commonly confused with

- **regression-testing** — Regression testing is the act of re-running against the set; the golden dataset is the set itself.
- **offline-vs-online-eval** — A golden dataset powers offline eval; online eval measures live traffic instead of a fixed set.

## Related

- eval-driven-development
- regression-testing
- llm-as-judge
- offline-vs-online-eval

---

Source: Claude Cert Prep — an independent, unofficial CCAR-P study resource (domain P4). Not affiliated with Anthropic.