LLM-as-a-Judge
Using a strong model with a rubric to score other models’ outputs, at scale.
LLM-as-a-judge replaces brittle string matching for open-ended outputs: a capable model grades responses against a clear rubric. Done well it tracks human judgement cheaply; watch for position, verbosity, and self-preference bias, and validate the judge against human labels.
Worked example: using a strong LLM to score or compare outputs against a rubric, instead of exact-match metrics — practical for open-ended tasks (summaries, chat) where there is no single right string. Gotcha: judges have biases — they favor longer answers, their own model family’s style, and the first option in a pairwise test (position bias); mitigate by randomizing order, calibrating against human labels on a sample, and using it for RELATIVE comparison rather than trusting an absolute 1-10 score.