# Observability & Tracing

Instrument each step of an agent/RAG run — prompts, retrievals, tool calls, tokens, latency — so you can debug and evaluate it.

## Why the CCAR-P exam tests this

Multi-step agent and RAG systems are non-deterministic, and P3 expects you to know how to make their behavior inspectable and measurable.

## Definition

**Observability** captures what happened inside an LLM run. A **trace** represents one end-to-end request; it is a tree of **spans**, where each span is a discrete step — a retrieval, a rerank, an **MCP tool call**, a model generation — recording inputs, outputs, token counts, latency, cost, and errors. This turns an opaque agent loop into something you can replay and diagnose.

Tracing is what lets you separate the failure layers you already care about: was it **retrieval** (wrong chunks came back) or **generation** (faithful answer over the right chunks)? Span-level data lets you attach the right metric to the right step — retrieval precision/recall on the retrieval span, faithfulness on the generation span — instead of only judging the final text.

In production, pair tracing with **evals** (offline and online), monitoring of latency/cost/error and tool-call success rates, and alerting on regressions. Be deliberate about logging sensitive retrieved content: traces can themselves become a data-exposure surface, so apply the same access controls to your telemetry.

## Exam trap

Judging only the final answer hides where a multi-step run broke — without span-level tracing you can't tell a retrieval failure from a generation failure, which are fixed differently.

## Commonly confused with

- **rag-pipeline** — Tracing observes the pipeline; it is measurement, not a retrieval stage that changes the answer.
- **mcp** — Tracing records MCP tool calls as spans; MCP performs them. Telemetry vs transport.

## Related

- rag-pipeline
- mcp
- data-layer-acls

---

Source: Claude Cert Prep — an independent, unofficial CCAR-P study resource (domain P3). Not affiliated with Anthropic.