Observability & Tracing for Claude Agents, Explained

What to instrument: spans across agent and tool hops

The unit of observability is the span: a single timed operation with a start, an end, and structured attributes. A request trace is the tree of spans produced by one user turn. In an agent, the natural spans are the model calls, each tool execution, each retrieval, and each sub-agent invocation. Give every span a parent so a tool call nests under the model turn that requested it, and stamp them all with one trace id so the whole request is reconstructable end to end.

• Model-request span: the exact rendered prompt, the model id, the stop reason, and the response.
• Tool-use span: the tool name, the input arguments the model produced, the result returned, and whether it errored.
• Retrieval span: the query, how many documents came back, and their ids or scores.
• Sub-agent span: a child trace for delegated work, linked back to the parent so delegation is visible, not hidden.
• Orchestration span: the glue between hops — routing decisions, retries, context assembly, and compaction.

What to log — and what to redact

Traces are only useful if they carry the payloads: the prompt actually sent (after your templating and context assembly), the raw model output, and each tool's inputs and outputs. But those payloads routinely contain user data — names, emails, account numbers, uploaded documents. Redact personally identifiable information at the point of capture, before it is written to your tracing backend, not after.

• Log the rendered prompt, not just the template — bugs hide in the interpolation, and caching depends on the exact bytes.
• Log tool arguments and results verbatim, since a malformed argument is the most common orchestration bug.
• Redact PII with a deterministic masking pass so the same value maps to the same token across spans and you can still correlate.
• Never log secrets, API keys, or credentials that pass through tool inputs — filter them at capture.
• Record the model id and configuration (effort, thinking, tools enabled) so you can reproduce the exact call later.

Token, cost, and latency per span

Every model-request span should carry its usage breakdown so cost and latency are attributable to the exact call that incurred them. The API returns this on each response: input tokens, output tokens, and cache activity. Attach it to the span, derive cost from the model's per-token price, and you can see which hop in a ten-step agent run is burning the budget — and whether your prompt cache is actually being read.

# Attach usage to each model-request span
span.set_attributes({
    "llm.model": response.model,
    "llm.input_tokens": response.usage.input_tokens,
    "llm.output_tokens": response.usage.output_tokens,
    "llm.cache_read_input_tokens": response.usage.cache_read_input_tokens,
    "llm.cache_creation_input_tokens": response.usage.cache_creation_input_tokens,
    "llm.stop_reason": response.stop_reason,
    "llm.latency_ms": elapsed_ms,
})

# input_tokens is the UNCACHED remainder only. Total prompt size =
#   input_tokens + cache_creation_input_tokens + cache_read_input_tokens
# If cache_read stays 0 across identical prefixes, a silent invalidator
# (a timestamp, a UUID, an unsorted tool list) is breaking the cache.

• Latency per span isolates the slow hop — often a tool call or retrieval, not the model.
• Tokens per span show where context is bloating; a runaway system prompt inflates every downstream turn.
• Cost per span turns an opaque monthly bill into a per-feature, per-user attribution you can act on.
• Cache-read tokens per span tell you whether caching is working; zero reads on a stable prefix is a bug to chase.

Production telemetry: drift, refusals, and tool errors

Development traces catch bugs; production telemetry catches the slow rot. Track rates over time, not just individual failures. A refusal rate that climbs after a prompt change, a tool-error rate that spikes when a downstream API degrades, or an output-length distribution that drifts all show up in aggregates long before any single user complains.

Sampling: you cannot store every trace at scale

Full-fidelity tracing of every request is affordable in development and expensive in production. At volume, storing every prompt and every tool payload for every request costs real money and slows the pipeline. The standard answer is sampling: keep a representative fraction of traces in full, while always keeping the ones that matter.

• Head sampling: decide at the start of a trace whether to keep it, at a fixed percentage — cheap but blind to outcome.
• Tail sampling: buffer the trace and keep it based on what happened — always retain errors, refusals, and slow requests.
• Always keep the tails: a 100% sample of failures and a small sample of successes gives you debuggability without the full bill.
• Keep aggregate metrics at 100% even when you sample full traces — you need every refusal counted even if you store few in full.

Model failure vs. orchestration failure

This is the distinction observability exists to make. A model failure is the model producing a bad output when it was given good context — a wrong answer, a hallucination, a malformed tool call from a well-formed schema. An orchestration failure is everything around the model: the wrong tool was selected, the retrieval returned nothing, the context was assembled incorrectly, a downstream API errored, or the glue code dropped a tool result. Both surface as "the agent gave a bad answer," and only the trace tells them apart.

Evaluating in production

Offline evals on a fixed test set tell you how the agent does on cases you thought of. Production is full of cases you did not. Evaluating in production means scoring real traffic — with automated checks, an LLM-as-judge, or human review on a sample — and feeding the results back as telemetry, so quality is a tracked metric alongside cost and latency rather than something you only measure before a release.

• Automated checks on every response: is the JSON valid, did the required tool run, is the answer non-empty and on-topic.
• LLM-as-judge on a sample: a separate model scores helpfulness or correctness against a rubric, and the score becomes a metric.
• Human review on the tails: route flagged, refused, or low-scoring traces to a queue for labeling.
• Close the loop: production evals surface failure modes that become new offline test cases, so the suite grows from real traffic.