CCDV-F Domain 4: Eval, Testing, and Debugging

Every request through a Claude application passes through the same four layers, in order: the API call itself, any tools the model invokes, the prompt and context the model was given, and finally the model's own reasoning over all of that. A failure can originate at any one of these layers, but by the time it surfaces to a user or a log line, it often looks identical to a failure at a different layer. The exam-relevant skill is working backward from the symptom to the correct layer, not guessing based on which layer is easiest to blame.

• API-layer failures: auth errors, rate limits, malformed requests, or server-side errors. These are identifiable from the HTTP status code and error type in the response — not from anything about what Claude said, because at this layer Claude never got a chance to say anything.
• Tool-execution failures: the tool itself errored, returned wrong or stale data, or timed out. Claude only knows what the tool told it — if the tool is broken, no amount of prompt engineering will fix the output, because the model is reasoning correctly over bad input.
• Prompt/context problems: Claude wasn't given what it needed — missing context, ambiguous instructions, or context that rotted or drifted over a long session. The model is working correctly; the input it received was insufficient or unclear.
• Genuine model-output problems: Claude had everything it needed — correct tool results, clear instructions, sufficient context — and still produced a wrong or poorly-reasoned answer. This is the rarest category to correctly diagnose, and the one most often mis-blamed for failures that actually belong to one of the other three layers.

Because the same surface symptom can map to multiple layers, the practical skill is checking evidence in a fixed order — status code and error type first, then tool_result blocks, then the prompt/context that was actually sent, and only then the model's reasoning itself. The table below is the shape of that check, mapped to typical exam-style scenarios.

Note the ordering: the table is deliberately sequenced from easiest-to-verify to hardest-to-verify. Status codes and error types are unambiguous and cheap to check. is_error on a tool_result is a structured, explicit signal. Prompt/context problems require reading what was actually sent, which takes more effort but is still mechanical. Genuine model-output problems require first confirming the other three layers are clean — which is exactly why that category is the rarest to correctly diagnose and the easiest to reach for prematurely.

Two structured fields carry most of the diagnostic weight in this domain, and both are frequently under-used in application logging: stop_reason on the response, and is_error on a tool_result content block.

stop_reason tells you why Claude stopped generating, and its possible values are themselves diagnostic categories rather than a single generic "done" flag: end_turn (a normal, complete response), max_tokens (the response was truncated by the token budget, not by the model deciding it was finished), stop_sequence (a configured stop sequence was hit), tool_use (Claude is pausing to call a tool and expects a tool_result back), pause_turn (a long-running turn paused and can be resumed), and refusal (the model declined to continue for policy reasons). Reading this field before reading the content of the response itself narrows the diagnosis immediately — a short, seemingly-incomplete answer with stop_reason: "max_tokens" is a token-budget problem with a mechanical fix, not evidence the model reasoned poorly.

is_error: true on a tool_result block is the specific, structured signal for a tool-layer failure. A well-built application surfaces this distinctly in its own logging and tracing rather than lumping every "the answer was wrong" case into one undifferentiated bucket. If tool errors and model-reasoning errors are logged identically, isolating the failing layer after the fact becomes guesswork instead of a lookup.

Once a failure is correctly isolated and fixed, the next question this domain tests is how you know it's fixed — and stays fixed. That's an evals and testing question, and it splits into two complementary approaches.

• Deterministic tests — exact-match comparisons, schema validation, code-execution checks — apply wherever the task has a checkable, well-defined correct answer. Use these wherever the task allows a deterministic check; they're cheap, fast, and unambiguous.
• Model-based (LLM-as-judge) evaluation is for genuinely open-ended output where no deterministic check is possible — quality, tone, helpfulness, or correctness judgments that can't be reduced to an exact match or a schema. Reserve this approach for cases that actually need it, since it's more expensive and less precise than a deterministic check.

The general rule the exam expects: prefer a deterministic check whenever the task structure permits one, and fall back to model-based evaluation only for the open-ended remainder. Using an LLM judge where a schema check or exact-match test would suffice adds cost and noise without adding rigor.

Tracing and observability are what make layer-isolation possible after the fact, rather than only in the moment something breaks. Logging enough structured detail per request — which layer executed, what each tool actually returned, what stop_reason was on each turn, and latency per step — is what lets a debugging session start from evidence instead of speculation. A system that only logs the final answer cannot be debugged when something goes wrong later, because every layer's evidence except the last one is already gone.

A support ticket comes in: a Claude-powered assistant told a customer their order had shipped when it had not. The surface symptom is identical to a genuine model-reasoning failure — the answer was simply wrong — so the instinct is often to rewrite the system prompt to be more careful about order status. That instinct skips the isolation step.

1. 01Check the API layer first: was the request itself successful? Status 200, no auth or rate-limit error. Rule out the API layer.
2. 02Check the tool layer next: the assistant called an order_status tool for this request. The trace shows the tool_result block for that call — and it carries is_error: false, but the returned payload shows status: "shipped" for an order that, per the shipping provider's own record, had not shipped yet.
3. 03Stop here. This is a tool-execution failure, not a model failure — the tool returned stale or incorrect data, and Claude reported exactly what the tool told it. The model reasoned correctly over bad input.
4. 04Confirm by checking the prompt/context layer anyway, since it's cheap to rule out: the system prompt correctly instructed the assistant to report tool data as-is and flag uncertainty; no ambiguity contributed here.
5. 05The fix is at the tool layer: investigate why order_status returned shipped prematurely — likely a caching or sync delay in the underlying order system — not a rewrite of the assistant's system prompt.