# Offline vs. Online Evaluation

Offline eval scores a fixed dataset before shipping; online eval measures real behavior on live traffic.

## Why the CCAR-P exam tests this

The exam tests knowing that passing offline evals does not guarantee real-world quality, so both are needed.

## Definition

**Offline evaluation** runs your system over a fixed `golden-dataset` in a controlled environment, before deployment. It is repeatable, cheap to rerun, and ideal for `regression-testing` and comparing candidate versions — but it only measures what your dataset happens to cover, and real inputs drift away from it over time.

**Online evaluation** measures the system on **live traffic** in production. It uses A/B tests, guardrail metrics, user signals (thumbs, retries, escalations), and sampled human or `llm-as-judge` review of real sessions. It catches distribution shift and emergent failures that no fixed set predicted, at the cost of being slower, noisier, and affecting real users.

The two are complementary, not interchangeable. Offline eval is your gate to ship; online eval is your check that the offline gate was right. A change that wins offline can still lose online, so promote gradually (canary, then ramp) and keep watching live metrics after launch.

## Exam trap

Passing every offline eval does not prove production quality. Skipping online monitoring lets distribution shift and real-user failures go unseen.

## Commonly confused with

- **regression-testing** — Regression testing is usually an offline gate; online eval adds live-traffic signals it can't capture.
- **golden-dataset** — The golden dataset is the fixed set offline eval uses; online eval measures live traffic instead.

## Related

- golden-dataset
- regression-testing
- eval-driven-development
- cost-latency-optimization

---

Source: Claude Cert Prep — an independent, unofficial CCAR-P study resource (domain P4). Not affiliated with Anthropic.