The check and the judge disagree

A team had both tools from the last two lessons. A substring check on the order number, and a calibrated judge on "does the reply address the problem". Then they wired them into one score. Each case got the mean of its scorers: the check at 0 or 1, the judge at 0 or 1, and a third scorer for tone. A case shipped if its mean was 0.6 or higher.

The run below is what happened next. The reply quoted order 82813 for a ticket about 88213. The check failed: 0. The judge passed: the reply named the double charge, promised a refund, sounded sincere. Tone passed. Mean 0.67. Shipped. A customer got a sincere apology about an order that is not theirs.

The check and the judge measure different kinds of truth. The check measures a fact: these digits are or are not in that text. The judge measures a judgment: this reply does or does not do the job. A fact that fails cannot be outvoted by two judgments that pass, because the judgments were made without knowing the fact. The judge was never asked "is the order number right". It was asked "does this reply address the problem", and from inside the reply the answer looked like yes.

Averaging hid this. A mean treats every scorer as the same kind of evidence. It is the eval equivalent of the double charge prompt itself: three jobs in one number, and when the first one is wrong the others cannot save it.

Check fails, judge passes. The one above. The check wins. Facts are vetoes. Nothing with a failed factual check ships, whatever the judge says. Treat the judge's pass as a note that the reply would have been fine if the fact were right.

Check passes, judge fails. The reply has the right number and the judge still says it does not do the job. This is the useful one. Read the case. Either the judge found something no check covers, such as a reply that answers a different question, or the judge is wrong, which lesson 5.2 taught you to measure. If the judge is right, ask whether the failure can be turned into a check. "The reply must mention a next action" can become a regex for a verb list. Many judge failures become checks over time, and the judge is left with the cases that need reading.

Both pass, the customer complains. Neither tool was asking the right question. Shankar and colleagues called this criteria drift in 2024: as people grade outputs, their idea of what good means changes, and the rubric written last month no longer matches. Hamel's field guide gives the cure: read the production traces, every week, and let the new failure categories write the next rubric line and the next check.

Two tiers, not one number. Tier one is the factual checks, each one a gate: any fail is a fail. Tier two is the judges, run only on cases that passed tier one, reported per rubric, never averaged with tier one. The summary line of an eval run then reads "38 of 40 passed the checks; of those, 35 passed the judge" and the two numbers mean different things.

Keep a list of disagreements as a first class output of every run. They are the most valuable rows in the table. A case where the check and the judge disagree is either a bug in the judge, a missing check, or a drift in the rubric, and each of those is a task someone should do this week.

"Checks are vetoes, judges are opinions, and I never average them. The summary has two numbers. When they disagree I read the case: if the judge was right, I try to turn it into a check; if the check was right, the judge rubric was asked the wrong question." Then tell the story of the sincere apology to the wrong customer, if you have one. Everyone does.