# Eval-Driven Development

Define measurable success criteria and an eval harness before building, then let scores gate every change.

## Why the CCAR-P exam tests this

The exam checks whether you define metrics before shipping rather than tuning prompts by vibes.

## Definition

**Eval-driven development (EDD)** flips the usual order of work: you write the **evaluation** first, then build the system to pass it. Before touching a prompt or a retriever, you decide what "good" means, encode it as a scored test set, and only then iterate. Every prompt tweak, model swap, or retrieval change is judged by whether the aggregate score moves up — not by whether a single cherry-picked example looks better.

A workable EDD loop has four parts: a **golden dataset** of representative inputs with expected outputs or rubrics, one or more **metrics** (exact-match, rubric score, `LLM-as-judge`, retrieval recall), a **runner** that executes the current system over the set, and a **threshold** that decides pass/fail. Changes ship only when the score holds or improves.

The discipline matters most because LLM output is stochastic: a change that fixes one case often silently breaks three others. Defining metrics **before** building prevents you from redefining "success" after the fact to match whatever the model happened to produce.

## Exam trap

EDD is not "add tests at the end." The metric and dataset must exist before you build, or you will unconsciously fit the definition of success to the output you already have.

## Commonly confused with

- **regression-testing** — Regression testing guards against drops on an existing suite; EDD is the broader practice of leading with metrics from day one.
- **golden-dataset** — The golden dataset is one artifact EDD depends on, not the whole methodology.

## Related

- golden-dataset
- llm-as-judge
- regression-testing
- offline-vs-online-eval

---

Source: Claude Cert Prep — an independent, unofficial CCAR-P study resource (domain P4). Not affiliated with Anthropic.