CCAR-P Domain 4: Evaluation, Testing & Optimization

An evaluation is a repeatable measurement of whether the system does the job. It starts with a golden dataset: a curated, versioned set of representative inputs paired with expected outcomes or acceptance criteria. Quality of the eval is bounded by quality of this dataset, so it must cover the real distribution of traffic — including the messy, ambiguous, and adversarial cases — not just the happy path an engineer typed while building the feature.

Metrics must be task-specific. A classification task can use precision, recall, and F1 against labels. An extraction task can use exact-match or field-level accuracy. An open-ended generation task has no single right answer, so you define a rubric and score against it. Choosing the wrong metric — for example, exact-match on a summarization task — produces a number that looks rigorous but measures nothing useful.

Offline evals run against your golden dataset in CI or a notebook, before shipping. They are fast, cheap, reproducible, and gate deployment. Online (production) evals measure the live system on real traffic: capturing outcomes, sampling for human or automated review, and watching business KPIs. Both are required — offline catches known-failure regressions, online catches the distribution shift and edge cases your dataset never anticipated.

• Offline: deterministic gate in CI, runs on every prompt/model change, blocks merge on regression.
• Online: samples live outputs, tracks real user outcomes, feeds new failure cases back into the golden dataset.
• Shadow / mirror: run a candidate on live traffic without serving its output, to compare before exposing users.
• The golden dataset is a living asset — production failures should be triaged and promoted into it so the same bug never regresses twice.

For open-ended output, human grading is the gold standard but does not scale. Using Claude to grade Claude's output — LLM-as-judge — scales to thousands of samples cheaply. It works only if you engineer it deliberately: give the judge a precise rubric with explicit criteria and a defined scale, ask for a short rationale before the score (reasoning improves reliability), and where possible provide a reference answer. A vague "rate this 1-10" produces noisy, non-reproducible scores.

Judges carry known biases you must control for: position bias (favoring the first option in a pairwise comparison — mitigate by swapping order and averaging), verbosity bias (rewarding longer answers), and self-preference. Calibration is the safeguard: periodically compare judge scores against a human-labeled sample and measure agreement. If the judge disagrees with humans, fix the rubric before trusting the judge at scale.

Prompts and agent definitions are code and deserve version control, tests, and CI. A regression test suite runs the golden dataset against a candidate prompt/model and fails the build if quality drops below a threshold. Because model output is non-deterministic, thresholds are statistical (for example, pass rate must stay above 95%) rather than exact-match on every row. Pin the model version in tests so an unrelated model update does not silently move your baseline.

Every failure should emit structured error metadata rather than a free-text log line. Capturing the input, the output, an error category, the model/prompt version, latency, and token counts turns a pile of anecdotes into a queryable dataset. That metadata is what lets you say "tool-argument errors rose 12% after the v7 prompt" instead of "it feels worse" — and it is the raw material for the next round of improvement.

# Structured error record captured on every eval/production failure
error = {
    "trace_id": trace_id,
    "prompt_version": "checkout-v7",
    "model": "claude-<pinned-version>",
    "input": user_input,
    "output": model_output,
    "error_category": "tool_arg_invalid",   # controlled vocabulary, not free text
    "expected": golden_expected,
    "latency_ms": 1840,
    "input_tokens": 3120,
    "output_tokens": 210,
}
# Aggregate by error_category + prompt_version to prioritize the next fix.

Cost is driven by tokens and by which model tier processes them. Four levers matter most at the architecture level. First, prompt caching: mark large stable prefixes (system prompt, tool definitions, long documents) as cacheable so repeat requests reuse them. Cache reads are billed at a large discount versus base input tokens, while a cache write carries a modest premium — so caching pays off when the same prefix is reused many times within the cache lifetime. Second, model routing: send easy, high-volume requests to a cheaper, faster tier (Haiku-class) and reserve the most capable tier for genuinely hard work, optionally using a cheap classifier to route.

Third, the Message Batches API: for work that does not need an immediate answer — bulk evals, offline enrichment, backfills — submit requests as a batch for a 50% discount on input and output tokens, with results returned within 24 hours. Batches support the same features as standard calls, including tool use. Fourth, trim what you send and store: prune stale conversation history, summarize long context, and cap verbose tool output before it re-enters the prompt, since every token you carry forward is billed on every subsequent turn.

Latency has two flavors: total time to complete a response and time-to-first-token (perceived latency). Streaming does not reduce total tokens or total time, but it starts delivering output immediately, so users perceive the system as far faster — essential for any interactive UI. For independent sub-tasks (fanning out over many documents, calling several tools that do not depend on each other), issue requests in parallel rather than serially so wall-clock time is bounded by the slowest call, not the sum.

Prompt caching also cuts latency, not just cost, because cached prefixes skip re-processing. Model routing helps here too: smaller models return faster. And the biggest structural lever is often shrinking the work — fewer tokens in and out, fewer sequential agent steps, and capping generation length all reduce time directly. Optimize the tail (p95/p99) that users actually feel, not just the average.

• Streaming: improves time-to-first-token and perceived speed; total generation time is unchanged.
• Parallelism: run independent model/tool calls concurrently to bound wall-clock time by the slowest.
• Prompt caching: cached prefixes cut both latency and cost on repeated large context.
• Cheaper/smaller model tier: faster responses for tasks that do not need the top tier.
• Trim output: cap max_tokens and avoid needlessly verbose formats to shorten generation.

Reliability is measured, not asserted. Track task success rate, error rate by category, and consistency across repeated runs of the same input (non-determinism means a single pass is not proof). For agents, measure at the trajectory level — did each step take a sensible action — not only whether the final message looked plausible. Set explicit reliability SLOs and alert when they degrade.

When you change a prompt, model, or agent, roll it out safely. A canary sends a small percentage of live traffic to the new version while the online evals watch its metrics; if quality or cost regresses, roll back before most users are affected. An A/B test splits traffic between two versions and compares outcomes with enough volume to reach statistical significance. Both are the production-side complement to offline regression tests: offline proves the candidate does not regress known cases, canary/A-B proves it holds up on real traffic.