LLM-as-Judge

LLM-as-judge uses a capable model to grade another model's output against a rubric or to pick the better of two responses (pairwise comparison). It scales evaluation for open-ended tasks — summaries, chat, code explanations — where there is no single correct string to match against.

Judges have well-known systematic biases you must design around. Position bias: in pairwise mode the judge favors whichever answer is shown first, so you swap order and average. Verbosity bias: longer answers are rated higher regardless of quality, so rubrics must reward correctness over length. There is also self-preference (a model rating its own family's style highly) and sensitivity to superficial formatting.

Because of these biases, a judge must be calibrated against human labels before you trust its numbers: score a sample both ways, measure agreement, and adjust the rubric until they align. Use a fixed, explicit rubric, request a short justification before the score to reduce noise, and prefer discrete labels over a raw confidence number. For bulk grading runs, the Batch API processes jobs asynchronously at 50% off with results in ≤24h.