# Cost & Latency Optimization

Cut spend and response time without dropping below your quality bar, using evals to guard the tradeoff.

## Why the CCAR-P exam tests this

The exam checks that you optimize cost and latency against measured quality, not by guesswork that quietly degrades output.

## Definition

**Cost and latency optimization** reduces the price and wall-clock time of serving an LLM system while keeping quality above a defined threshold. The core levers: route simpler requests to **smaller/cheaper models** and reserve the strong model for hard cases; add **prompt caching** to avoid re-paying for stable context; tighten prompts and cap output tokens; and **stream** responses to cut perceived latency.

For non-interactive, high-volume work — bulk grading, dataset labeling, offline scoring — the **Batch API** is the big win: about **50% cheaper** with results returned **within 24 hours**, since you trade immediacy for throughput. This is ideal for running large `llm-as-judge` eval jobs where a day of turnaround is fine.

Every optimization is a **tradeoff against quality**, so it must be validated with your eval suite, never assumed. Track cost-per-request and latency percentiles (p50/p95/p99) alongside quality scores, and only ship a cheaper configuration when `regression-testing` confirms scores held. Distinguish **latency** (per-request time) from **throughput** (requests per unit time) — batching improves throughput and cost while *increasing* per-request latency.

## Exam trap

Cheaper and faster is not free. Any optimization can silently degrade quality, so it only counts if the eval suite confirms the score held.

## Commonly confused with

- **offline-vs-online-eval** — Latency and cost show up as online production metrics; this concept is the practice of improving them against a quality bar.
- **llm-as-judge** — The Batch API cuts cost for bulk judge runs; the judge itself is the scoring method, not the optimization.

## Related

- llm-as-judge
- regression-testing
- offline-vs-online-eval
- eval-driven-development

---

Source: Claude Cert Prep — an independent, unofficial CCAR-P study resource (domain P4). Not affiliated with Anthropic.