# LLM-as-Judge

Use a strong LLM to score or compare outputs against a rubric when exact-match grading is impossible.

## Why the CCAR-P exam tests this

The exam probes whether you know LLM judges carry systematic biases and must be calibrated, not trusted blindly.

## Definition

**LLM-as-judge** uses a capable model to grade another model's output against a **rubric** or to pick the better of two responses (pairwise comparison). It scales evaluation for open-ended tasks — summaries, chat, code explanations — where there is no single correct string to match against.

Judges have well-known **systematic biases** you must design around. **Position bias**: in pairwise mode the judge favors whichever answer is shown first, so you swap order and average. **Verbosity bias**: longer answers are rated higher regardless of quality, so rubrics must reward correctness over length. There is also **self-preference** (a model rating its own family's style highly) and sensitivity to superficial formatting.

Because of these biases, a judge must be **calibrated** against human labels before you trust its numbers: score a sample both ways, measure agreement, and adjust the rubric until they align. Use a fixed, explicit rubric, request a short justification before the score to reduce noise, and prefer discrete labels over a raw `confidence` number. For bulk grading runs, the **Batch API** processes jobs asynchronously at **50% off** with results in **≤24h**.

## Exam trap

An LLM judge's score is not ground truth. Ignoring position and verbosity bias, or skipping calibration against human labels, makes the numbers confidently wrong.

## Commonly confused with

- **golden-dataset** — A golden dataset supplies the inputs and references; LLM-as-judge is one scoring method applied over it.
- **retrieval-vs-faithfulness** — A judge can measure faithfulness, but faithfulness is the property being measured, not the grading mechanism.

## Related

- golden-dataset
- eval-driven-development
- retrieval-vs-faithfulness
- cost-latency-optimization

---

Source: Claude Cert Prep — an independent, unofficial CCAR-P study resource (domain P4). Not affiliated with Anthropic.