ExplainerCCAR-P · P69 min read

Incident Response for Claude Systems, Explained

How to handle a production incident in an LLM system the way you'd handle any outage: detect, contain, diagnose, roll back, and learn, from a runbook you wrote before the pager went off.

The short answer

Handle a production incident in a Claude system the way you handle any outage: detect it through observability, contain the damage by failing closed or disabling the risky path, diagnose whether the cause is the model, the orchestration, or the data, roll back to a known-good state, and run a blameless postmortem. The goal is a pre-agreed runbook you execute under pressure, not improvisation invented during the outage.

Every production system fails eventually, and systems built on a large language model fail in some ways ordinary software does not: the output can be confidently wrong, a tool can return garbage the model then trusts, a prompt-injection payload can steer the agent, or a runaway loop can burn through your budget in minutes. What makes these incidents survivable is not heroics at 2 a.m. It is having decided, calmly and in advance, what counts as an incident, how you will detect it, how you will stop the bleeding, and how you will get back to a known-good state.

This article walks the incident lifecycle for a Claude-based system: recognizing what an incident looks like, detecting it through observability, containing it by failing closed, diagnosing it by mapping the symptom to the architectural cause, rolling back, and closing the loop with a blameless postmortem. The through-line is that the response should be a runbook you execute, not a plan you invent while the system is on fire. Improvisation under pressure is how a small regression becomes a large outage.

What an incident looks like

The first job is recognizing that you are in an incident at all. In an LLM system the failure is often not a crash or a stack trace. The service returns HTTP 200, the tokens flow, and the output is wrong, harmful, expensive, or subtly off. Because nothing errors, these failures can run for a long time before anyone notices, which is exactly what makes naming them in advance so valuable.

Bad outputs: the model returns answers that are wrong, fabricated, or off-policy, while the request itself succeeds and logs look clean.
Tool failures: a tool or downstream API returns errors, empty results, or malformed data, and the model reasons on top of it as if it were valid.
Cost spikes: a retry storm, a runaway agent loop, or an unexpectedly long context drives token spend far above baseline in a short window.
Safety regressions: a change causes the system to produce content it should have refused, or to refuse legitimate requests it used to handle.
Prompt injection: untrusted content in a document, web page, or tool result contains instructions that hijack the agent's behavior.

Detection through observability

You cannot respond to what you cannot see. Because so many LLM failures are silent, detection depends on instrumentation that watches outcomes and rates rather than just uptime. The goal is to shorten the time between 'the system started misbehaving' and 'someone knows,' because that interval is the size of the damage.

Log the shape of every interaction: inputs, the served output, which tools were called with what, token usage, latency, and stop or finish reasons, with request IDs that let you trace one flow end to end.
Track rates against a baseline: refusal rate, tool-error rate, cost per request, output length, and any automated quality proxy you have. Alert on deviations, not just on zeroes.
Alert on the silent signals specifically: a cost-per-hour ceiling, a spike in a specific tool's error rate, a jump in refusals, a collapse in average output length.
Keep enough context to reproduce: the exact prompt, model version, and tool results for a flagged request, so diagnosis starts from evidence rather than guesswork.

Containment: fail closed and disable the risky path

Once you know something is wrong, the first move is not to diagnose it. It is to stop it from getting worse. Containment is about limiting blast radius while you still do not fully understand the cause, and the guiding principle is to fail closed: when the safe answer is unknown, default to the safe outcome, which is usually 'do not proceed.'

Fail closed on the affected control: if a screening check, a tool, or a downstream service is misbehaving, block or hold the action rather than letting it through unguarded.
Disable the risky path: turn off the specific tool, feature flag, or capability that is causing harm, ideally without taking the whole system down. A degraded system that refuses the dangerous action beats a fully-available one that keeps doing damage.
Cut the loop or the spend: if the incident is a runaway agent or a cost spike, kill the loop and clamp the budget before you investigate.
Quarantine untrusted input: if injection is suspected, stop treating the tainted source as trusted so the payload stops steering the agent.

Diagnosis: map the symptom to the architecture

With the bleeding stopped, you can diagnose. The most useful discipline here is to resist blaming 'the model' by reflex. Most LLM incidents are not the model being dumb; they are the orchestration feeding it bad context, a tool contract that misleads it, or data that changed underneath it. The skill is mapping the observed symptom to the layer that actually produced it: model, orchestration, or data.

Symptom	Most likely architectural cause	Where to look
Confidently wrong answer	Dropped, truncated, or over-summarized context	The context assembly and any compaction or trimming step
Wrong tool chosen, or tool overused	A misleading or over-eager tool description	The tool definitions and their trigger wording
Model trusts bad data	Untrusted content not tagged or quarantined	Input screening and source-trust handling
Sudden cost or latency spike	Retry storm or runaway loop in orchestration	Loop bounds, retry logic, and context growth
Behavior changed with no code change	A repointed model alias or a changed upstream data source	Model version pinning and data freshness

Rollback to a known-good state

Diagnosis tells you what broke; rollback gets you back to working while you fix it properly. The fastest resolution for most incidents is to revert whatever changed to the last known-good state, rather than trying to hot-fix forward under pressure. That requires having designed the system so that every risky element, the model version, the prompt, the tool set, the config, can be reverted independently and quickly.

Identify what changed most recently: a prompt edit, a model upgrade, a tool addition, a config flag, or an upstream data change. Recent change is the first suspect.
Revert that change to the known-good version. If you cannot tell what changed because nothing is versioned, that gap is itself a top finding for the postmortem.
Prefer reverting to hot-fixing forward during the incident. A fix invented under pressure is a fresh, untested change; a rollback returns you to a state you have already validated.
Confirm recovery against the same signals that flagged the incident, so you know the rollback actually resolved it rather than masking it.

Runbooks and escalation

Everything above works far better when it is written down in advance. A runbook is the pre-agreed script for an incident: it says what the alerts mean, what to check first, how to contain each class of failure, how to roll back, and who to call when. The value of a runbook is that it moves the thinking to a calm moment and leaves only execution for the stressful one.

Write a runbook per failure class: bad outputs, tool failure, cost spike, safety regression, suspected injection. Each should name the detection signal, the containment step, the rollback, and the verification.
Define escalation on policy, not on gut feel: fixed conditions such as 'safety regression in production' or 'cost over threshold for N minutes' page the right owner automatically, rather than leaving each responder to decide whether it is 'bad enough' to escalate.
Name owners and decision rights: who can disable a feature, who approves a rollback, who talks to affected stakeholders. Ambiguity here costs minutes you do not have.
Rehearse it. A runbook nobody has practiced is a document, not a capability.

The blameless postmortem

The incident is not over when service is restored. It is over when you have learned from it. A blameless postmortem reconstructs what happened, why the system allowed it, and what will change so it cannot recur the same way. Blameless means the focus is on the system and its gaps, not on finding a person to fault, because a culture that hunts for someone to blame is a culture where the next incident gets hidden rather than reported.

Build a timeline: when it started, when it was detected, when it was contained, when it was resolved, and what each gap between those points tells you.
Find the systemic cause, not the scapegoat: ask why the system permitted the failure and why detection took as long as it did, rather than who typed the change.
Turn findings into concrete actions: a missing alert to add, a fail-open control to make fail-closed, a runbook to write, a version to start pinning, each with an owner and a date.
Feed it back in: update the runbooks, the observability, and the review checklist so the next incident starts from what this one taught you.

Key takeaways

→The goal of incident response is to execute a pre-agreed runbook under pressure, not to improvise a plan while the system is failing.
→LLM incidents are often silent: bad outputs, trusted-but-bad tool data, cost spikes, safety regressions, and prompt injection all return success while doing harm, so define incidents by outcome, not just by errors.
→Detection depends on observability that watches outcomes and rates against a baseline, because you cannot respond to a silent failure you cannot see.
→Contain first by failing closed and disabling the risky path; a degraded system that blocks the dangerous action beats an available one that keeps doing damage.
→Diagnose by mapping the symptom to the architecture: confident-wrong usually means dropped or summarized context, and mis-routing usually means a misleading tool description, not a dumb model.
→Roll back to a known-good state rather than hot-fixing forward, which requires pinning versions and shipping changes behind flags so reverting is one fast action.
→Escalate on fixed policy conditions rather than on a responder's confidence, and close every incident with a blameless postmortem that turns gaps into owned, dated fixes.

Now practice it

Reading builds recognition; practice builds judgment. Try these on the P6 material.

Quick P6 warm-up Full P6 practice CCAR-P mock exam

Frequently asked

What counts as an incident in a Claude-based system?

Any outcome-level failure, not just a crash. That includes wrong or fabricated outputs, tools returning bad data the model then trusts, cost spikes from retry storms or runaway loops, safety regressions where the system produces or refuses the wrong things, and prompt injection steering the agent. Most of these return a successful HTTP response, so define incidents by outcome rather than by exceptions or your health checks will miss the ones that matter.

Why should I fail closed during an incident?

Because containment is about limiting harm before you understand the cause, and the safe default when the right answer is unknown is 'do not proceed.' Failing closed on the affected control blocks or holds the risky action; failing open bypasses the broken check so the harm keeps flowing at full volume. A blocked action is recoverable, a shipped harmful action often is not, so during an incident failing closed on the affected path is almost always correct.

How do I tell whether the model, the orchestration, or the data caused an incident?

Map the symptom to the layer. A confidently wrong answer usually means the orchestration dropped, truncated, or over-summarized the context the model needed. A wrong or overused tool usually means a misleading or over-eager tool description. A model trusting bad data usually means untrusted input was not tagged or quarantined. A behavior change with no code change usually means a repointed model alias or a changed upstream data source. Diagnose the architecture before blaming the model.

Why roll back instead of fixing the problem forward?

A rollback returns you to a state you have already validated, while a hot-fix invented during the incident is a fresh, untested change that can make things worse. The fastest path to a working system is usually to revert whatever changed most recently to its known-good version, then fix the root cause properly afterward. This only works if you designed for reversibility by pinning versions and shipping changes behind flags.

What makes a postmortem 'blameless' and why does it matter?

A blameless postmortem focuses on why the system allowed the failure and why detection was slow, not on which person to fault. It matters because a culture that punishes people for incidents teaches them to hide problems rather than report them, which destroys the fast, honest reporting that good incident response depends on. Keep the output focused on systemic gaps turned into concrete, owned, dated fixes to the runbooks, observability, and controls.

All explainers

Independent, unofficial study material from Claude Cert Prep. Not affiliated with Anthropic.