Observability & Tracing

Observability captures what happened inside an LLM run. A trace represents one end-to-end request; it is a tree of spans, where each span is a discrete step — a retrieval, a rerank, an MCP tool call, a model generation — recording inputs, outputs, token counts, latency, cost, and errors. This turns an opaque agent loop into something you can replay and diagnose.

Tracing is what lets you separate the failure layers you already care about: was it retrieval (wrong chunks came back) or generation (faithful answer over the right chunks)? Span-level data lets you attach the right metric to the right step — retrieval precision/recall on the retrieval span, faithfulness on the generation span — instead of only judging the final text.

In production, pair tracing with evals (offline and online), monitoring of latency/cost/error and tool-call success rates, and alerting on regressions. Be deliberate about logging sensitive retrieved content: traces can themselves become a data-exposure surface, so apply the same access controls to your telemetry.