Escalate on Policy, Not Confidence
Also: policy-based escalation · deterministic escalation · confidence is not probability
Route to a human on fixed policy rules, never on the model's self-reported confidence score.
Why CCAR-P tests this
Candidates must recognize that a model's stated confidence is not a calibrated probability and cannot be trusted as an escalation trigger.
A model that says it is 0.92 confident is not telling you there is a 92% chance it is correct. That number is a self-report generated by the same system that may be wrong — it is not calibrated, not a ground-truth probability, and is trivially high even when the model is hallucinating. Treating confidence >= 0.9 as "safe to auto-approve" is a classic control failure.
The correct pattern is to escalate on policy: define fixed, deterministic rules that decide when a human must be in the loop — for example, any action touching money above a threshold, any irreversible operation, any request matching a sensitive category, or any low-agreement result from an ensemble check. These triggers live in deterministic code, not in the model's judgment.
Confidence signals can inform a policy (e.g. as one input among several) but must never be the policy. The governing question is "does this action class require review?" — a property of the request and its blast radius — not "how sure does the model feel?"
Exam trap
A distractor will offer "auto-approve when the model's confidence exceeds 0.9." This is wrong because the confidence figure is uncalibrated self-report, not a real probability of correctness.
Commonly confused with
Independent, unofficial study material from Claude Cert Prep. Not affiliated with Anthropic.