# Regression Testing

Re-run a fixed eval suite on every change so quality drops are caught before they reach users.

## Why the CCAR-P exam tests this

The exam checks that you use an automated suite to prevent silent quality regressions from prompt or model changes.

## Definition

**Regression testing** re-runs a stable eval suite — built on a `golden-dataset` — against each candidate version and blocks the change if scores fall below a set **threshold**. Its whole job is to catch **silent quality drops**: the summary that got worse, the format that broke, the safety case that regressed, none of which are visible from spot-checking a few outputs.

LLM systems regress from causes traditional software doesn't have: a **model version** update, a prompt edit that helps one case and hurts others, a changed temperature, or a retriever tweak. Because outputs are stochastic, compare **aggregate** scores across the suite (and account for run-to-run variance) rather than reacting to any single example.

When a real bug reaches production, add a case reproducing it to the suite so it can never silently return. Wire the suite into CI as a **gate** on merges and deploys, and log per-objective/per-domain breakdowns so you see *where* quality moved, not just that it did.

## Exam trap

A regression suite catches drops only for what it covers. If new failure modes are never added as cases, they regress silently despite a green suite.

## Commonly confused with

- **eval-driven-development** — EDD is the broad metrics-first practice; regression testing is the specific safety net that reruns the suite per change.
- **offline-vs-online-eval** — Regression testing is typically an offline gate; it won't catch live-traffic issues online eval surfaces.

## Related

- golden-dataset
- eval-driven-development
- offline-vs-online-eval
- cost-latency-optimization

---

Source: Claude Cert Prep — an independent, unofficial CCAR-P study resource (domain P4). Not affiliated with Anthropic.