Eval-Driven Development

Eval-driven development (EDD) flips the usual order of work: you write the evaluation first, then build the system to pass it. Before touching a prompt or a retriever, you decide what "good" means, encode it as a scored test set, and only then iterate. Every prompt tweak, model swap, or retrieval change is judged by whether the aggregate score moves up — not by whether a single cherry-picked example looks better.

A workable EDD loop has four parts: a golden dataset of representative inputs with expected outputs or rubrics, one or more metrics (exact-match, rubric score, LLM-as-judge, retrieval recall), a runner that executes the current system over the set, and a threshold that decides pass/fail. Changes ship only when the score holds or improves.

The discipline matters most because LLM output is stochastic: a change that fixes one case often silently breaks three others. Defining metrics before building prevents you from redefining "success" after the fact to match whatever the model happened to produce.