Evaluating Claude: Evals & LLM-as-Judge, Explained

Start with the metric, not the model: eval-driven development

Eval-driven development flips the usual order. Before you write a prompt or wire up a tool, you write down what "good" means as something you can measure. If you cannot state the success criterion, you cannot tell whether any change helped — you are just moving prompts around and trusting your gut on a handful of cherry-picked examples.

Concretely, decide up front: what task is the system doing, what does a correct output look like, what failure modes matter most, and what score threshold is "ship-ready." These decisions become your eval harness. Every future change — a new model version, a reworded system prompt, an added tool — is then a hypothesis you test against that harness instead of a leap of faith.

• Write the success criteria as a sentence a non-engineer could check.
• Turn each criterion into a scorer (exact match, rubric, or judge).
• Fix a baseline score, then require every change to beat it.
• Treat the eval suite as production code — version it, review it, keep it green.

Designing an eval dataset you can trust

Your eval is only as honest as the examples in it. A golden set is a fixed, curated collection of inputs paired with known-good expected outputs (or grading rubrics). It should look like the traffic you actually expect — same distribution of easy and hard cases — plus deliberate coverage of the edge cases and failure modes you are most afraid of.

Aim for a set large enough that a one-example fluke does not swing the score, but curated enough that you can read failures by hand. Include adversarial inputs, ambiguous requests, and cases where the correct answer is "I don't know" or "I can't help with that." Freeze this set: if you keep editing it to match the model, you have lost your independent measuring stick.

• Representative cases — match real input distribution.
• Edge cases — long inputs, empty fields, wrong language, injection attempts.
• Negative cases — where the right move is to refuse or ask a clarifying question.
• Known-hard cases — the specific failures that prompted the eval in the first place.

Choosing metrics that mean something

There is no single "accuracy" number for most language tasks. The metric has to match the task's shape. Classification and extraction can use exact-match or F1 against a key. Open-ended generation needs a rubric or a judge. Retrieval systems need their own separate scores (covered below). Pick the cheapest scorer that actually captures what you care about.

LLM-as-judge: scaling grading without lying to yourself

For open-ended outputs, human grading does not scale and exact-match does not apply. LLM-as-judge uses a model (often Claude) to score another model's output against an explicit rubric. Done well, it correlates strongly with human judgment at a fraction of the cost. Done carelessly, it produces confident, consistent, and consistently wrong scores.

Design the judge like an exam rubric, not a vibe check. Give it a narrow question, a scale with described anchors, and the reference material it needs. Ask for a short justification before the score so the reasoning is inspectable. Prefer pairwise comparison ("is A or B better?") or low-cardinality scales (1–5) over asking for a spurious two-decimal number the judge cannot actually distinguish.

Judge prompt (sketch)
---------------------
You are grading a support answer for FAITHFULNESS only.
Context passages: <<retrieved docs>>
Answer to grade: <<candidate>>

Step 1: List each factual claim in the answer.
Step 2: For each claim, mark SUPPORTED / UNSUPPORTED / CONTRADICTED
        using ONLY the context passages.
Step 3: Score 1-5 using the anchors below, then explain in one sentence.
  5 = every claim supported; 1 = core claim contradicted.
Return JSON: {"claims":[...], "score": n, "reason": "..."}

• One rubric = one dimension. Grade faithfulness, helpfulness, and tone separately, not in a single blended number.
• Give the judge the ground truth or source context; do not make it recall facts from memory.
• Require reasoning before the score to expose bad judgments.
• Validate the judge against a human-labeled slice before you trust it at scale.

The trap: bias in the judge and confidence that isn't accuracy

Two failure modes fool teams constantly. First, LLM judges are biased. They tend to favor the first option presented (position bias), longer and more elaborate answers (verbosity bias), answers written in their own style (self-preference), and confident tone over correctness. If you A/B two systems by always putting the new one second, you may be measuring order, not quality.

Second, and more insidious: a model's raw confidence is not its accuracy. When a model outputs a 0.92 probability, that is a calibrated claim only if, across all the times it says 0.92, it is right about 92% of the time — and out of the box it usually is not. A 0.92 confidence and 92% accuracy are different quantities. You establish a usable threshold empirically: measure accuracy at each confidence level on your golden set, then pick the cutoff that hits your target. The threshold is earned from data, never read off the raw score.

• Counter position bias: randomize order, or grade both orders and average.
• Counter verbosity bias: instruct the judge to ignore length; penalize unsupported padding.
• Counter self-preference: where stakes are high, use a different judge model or human spot-checks.
• Counter miscalibration: build a reliability plot; treat confidence as a feature to calibrate, not a truth.

RAG: keep retrieval metrics separate from generation faithfulness

In a retrieval-augmented system, a wrong answer has two very different causes, and one blended "quality" score hides which one you have. Either retrieval failed to surface the right passage, or retrieval succeeded and generation still produced an unsupported answer. These need separate scores because they have separate fixes — better indexing/chunking versus better prompting/grounding.

Diagnostic rule of thumb: if faithfulness is high but end-to-end accuracy is low, your retrieval is feeding the model bad or missing context — fix the index. If retrieval recall is high but faithfulness is low, the model is ignoring or embellishing the context — fix the generation prompt and grounding.

Offline, online, canary, and regression: closing the loop

Offline evals run against your frozen golden set in CI — fast, cheap, deterministic enough to gate every change. But offline sets never fully capture live traffic, so pair them with online evaluation: sample real production interactions, judge them (LLM-as-judge or human), and watch the metric over time. A canary release routes a small slice of live traffic to the new version and compares its online scores against the current one before you roll out to everyone.

Finally, make evals a regression gate. Every bug you fix should become a permanent test case, so the same failure cannot silently return after a future prompt or model change. This is how an eval suite compounds in value: it accumulates the exact mistakes your system has made and refuses to let them recur.

• Offline: golden set in CI, blocks merges that drop below baseline.
• Online: sample and score live traffic to catch distribution shift.
• Canary: small live slice on the new version, compared before full rollout.
• Regression: every fixed bug becomes a locked-in test case forever.