Offline vs. Online Evaluation
Also: pre-deploy vs. production eval · batch eval vs. live eval · held-out vs. A/B
Offline eval scores a fixed dataset before shipping; online eval measures real behavior on live traffic.
Why CCAR-P tests this
The exam tests knowing that passing offline evals does not guarantee real-world quality, so both are needed.
Offline evaluation runs your system over a fixed golden-dataset in a controlled environment, before deployment. It is repeatable, cheap to rerun, and ideal for regression-testing and comparing candidate versions — but it only measures what your dataset happens to cover, and real inputs drift away from it over time.
Online evaluation measures the system on live traffic in production. It uses A/B tests, guardrail metrics, user signals (thumbs, retries, escalations), and sampled human or llm-as-judge review of real sessions. It catches distribution shift and emergent failures that no fixed set predicted, at the cost of being slower, noisier, and affecting real users.
The two are complementary, not interchangeable. Offline eval is your gate to ship; online eval is your check that the offline gate was right. A change that wins offline can still lose online, so promote gradually (canary, then ramp) and keep watching live metrics after launch.
Exam trap
Passing every offline eval does not prove production quality. Skipping online monitoring lets distribution shift and real-user failures go unseen.
Commonly confused with
Independent, unofficial study material from Claude Cert Prep. Not affiliated with Anthropic.