Regression Testing

Regression testing re-runs a stable eval suite — built on a golden-dataset — against each candidate version and blocks the change if scores fall below a set threshold. Its whole job is to catch silent quality drops: the summary that got worse, the format that broke, the safety case that regressed, none of which are visible from spot-checking a few outputs.

LLM systems regress from causes traditional software doesn't have: a model version update, a prompt edit that helps one case and hurts others, a changed temperature, or a retriever tweak. Because outputs are stochastic, compare aggregate scores across the suite (and account for run-to-run variance) rather than reacting to any single example.

When a real bug reaches production, add a case reproducing it to the suite so it can never silently return. Wire the suite into CI as a gate on merges and deploys, and log per-objective/per-domain breakdowns so you see where quality moved, not just that it did.