Golden Dataset
Also: gold set · eval set · reference dataset · ground-truth set
A curated, versioned set of representative inputs with trusted expected outputs used to score every change.
Why CCAR-P tests this
The exam checks that you build a stable reference set before optimizing, so scores are comparable over time.
A golden dataset is a fixed, hand-verified collection of inputs paired with expected outputs or grading rubrics. It is the yardstick every eval run measures against, so its quality caps the quality of your entire evaluation. Items should be representative of real traffic and deliberately include hard cases, edge cases, and known failure modes — not just easy happy-path examples.
Treat it like code: version it, review changes, and keep it stable so scores are comparable across runs. When you fix a new class of bug in production, add a case for it to the set so the fix is protected going forward. Coverage should span your objectives and domains rather than clustering on whatever was easy to collect.
A golden set is the shared substrate for eval-driven-development, regression-testing, and llm-as-judge — all of them run over it. Keep held-out slices so you can detect overfitting to the set itself, and avoid leaking the golden answers into prompts or fine-tuning data.
Exam trap
A golden set that only contains easy, happy-path examples inflates every score. Coverage of hard and failure cases is what makes it trustworthy.
Commonly confused with
Independent, unofficial study material from Claude Cert Prep. Not affiliated with Anthropic.