LLM-as-judge
Using one model as the grader for another model's answers when quality is too fuzzy for an exact test.
See it
What it is
LLM-as-judge means asking a model to score or compare model outputs against a written rubric. A pointwise judge grades one answer at a time; a pairwise judge picks between A and B. It fills the gap where exact checks cannot see quality, such as whether a summary is faithful, a reply follows the requested tone, or an explanation is useful.
Reach for it when a human can recognize a good answer but cannot reduce that judgment to string equality or a schema check. Give the judge the source material, explicit criteria, labeled examples, and permission to mark a case unscorable. For comparisons, hide model names and randomize which answer appears first.
Gotcha: the judge brings biases of its own. Models can favor longer answers, the first answer, familiar writing styles, or outputs resembling their own; the 2023 MT-Bench and Chatbot Arena paper that popularized the method measured those biases in the same breath as it proposed it. Calibrate scores against human labels, keep deterministic checks for facts a program can verify, and never treat a judge score as ground truth just because it has two decimal places.
Ask AI for it
Build a pairwise LLM-as-judge runner. Give the judge the original input, reference material, two anonymized candidate answers, and a rubric with separate correctness, completeness, relevance, and style criteria. Randomize A and B on every repeat, request output constrained by JSON Schema with winner, per-criterion scores, evidence, and cannot_judge_reason, and aggregate repeated comparisons with a Bradley-Terry model. Calibrate the result against a checked-in set of human labels, report agreement and position bias, and refuse to publish a score when judge-human agreement falls below the configured threshold.