Pick the rubric: an LLM judge, and how to calibrate it against your own labels

"Does the reply address the customer's actual problem?" No substring check answers that. A reply can mention the order number and the words "double charge" and still be useless. For this kind of question you grade with a model: give it the ticket, the reply and a rubric, and ask for a verdict. That is an LLM judge.

Judges work better than people expected. The MT-Bench paper from 2023 found that GPT-4 as a judge agreed with human raters over 80% of the time, the same rate at which the humans agreed with each other. It also named the ways judges fail: position bias (in pairwise grading the first answer wins more often), verbosity bias (longer looks better), and self enhancement bias (a model prefers its own writing). None of those go away by asking nicely. You design around them.

Four rules, from Hamel Husain's judge guide and Anthropic's eval docs. One, the verdict is pass or fail, not a score from 1 to 5. A 3 means nothing you can act on and two raters will not agree on what a 3 is. Two, the judge writes a short critique before the verdict, so you can see why it decided and so its reasoning is on the record. Three, the judge is a different model from the one being graded, to blunt self preference. Four, the rubric is written by the one person who knows what good looks like, the domain expert, in their words, with one or two examples of a pass and a fail.

A rubric for the support reply looks like this: "Pass if the reply names the customer's problem in its own words, states the next action and who takes it, and contains no claim the ticket does not support. Fail otherwise. Write one sentence of reasoning, then PASS or FAIL." Short. Nothing about tone, because tone was not the bug.

A judge you have not measured is a second opinion from a stranger. So before you trust it, label cases yourself. Take forty real replies, read each one with the rubric in hand, write pass or fail and a one line reason. Then run the judge on the same forty and compare.

Agreement rate is the easy number: matches divided by cases. In the calibration table below the judge matches the human on 34 of 40, so 85%. The catch is that agreement can be high by chance when most cases are passes. Cohen's kappa corrects for that: kappa equals (observed agreement minus chance agreement) divided by (1 minus chance agreement), where chance agreement comes from how often each rater says pass. If both say pass 80% of the time, chance agreement is 0.8 times 0.8 plus 0.2 times 0.2, which is 0.68, and 85% observed gives kappa of 0.53. Landis and Koch's bands call 0.41 to 0.60 moderate, 0.61 to 0.80 substantial, above 0.80 almost perfect. A judge at 0.53 is not ready.

Then look at the six disagreements one by one. In the table, case 23 is the pattern: the human failed a reply that was long, warm and wrong about the next step, and the judge passed it. That is verbosity bias, and the fix is a line in the rubric: "length and politeness do not count". Re-run, re-measure. Hamel's guide describes this loop, expert labels, judge, disagreements, rubric edit, until agreement is high and stays high. Keep ten of the forty labels out of the loop so you can check you did not tune the rubric to the labels.

"Pass or fail with a critique, a different model as judge, a rubric written by the expert. I labeled forty cases by hand first, measured agreement and kappa, read the disagreements, fixed the rubric, and re-measured. We trusted the judge at kappa above 0.7 and we re-check it every time the rubric or the judge model changes." The number they want to hear is the kappa, not the agreement rate.