The held out set: 40 cases from real transcripts, labeled by hand

Not from your head. From the trace store in lesson 5.0. Pull a sample of real runs from the last month, and read them. Hamel Husain's field guide calls this error analysis: read the trace, write one line on what went wrong, keep going until the lines start repeating, then group them. For the support agent the groups were: wrong order number, no order number in the ticket, two orders in one ticket, refund already issued, wrong language. Those groups are your case categories, and the counts tell you which ones matter.

Then sample cases by category, not at random. Random sampling gives you forty easy tickets and one hard one. Take eight from each of the five categories, so every category can move the number. Where production has no example yet of a category you know exists, write two or three by hand and mark them synthetic, so you remember which rows are real.

One person who knows the job. For a support agent that is the support lead, not the engineer. Each case gets the input, the expected output for the checks, and a pass or fail with a one line reason for the judge, in that person's words. Pass or fail, not a scale, for the reason lesson 5.2 gave. Anthropic's eval guide prefers many cheaply graded cases to a few expensive ones, and that is true for the dev set. For the held out set, forty hand labeled cases is the price of a number you can say out loud.

Labels drift. Shankar and colleagues showed in 2024 that graders change their own criteria as they grade. So the labeler writes the reason, not just the verdict, and the reasons get re-read when the rubric changes.

The dev set is the forty you look at every day. You tune the prompt against it, you add today's bug to it, you read its failures. The held out set is another forty, same categories, that you run only when you want to report a number, and never read while tuning. The wall between them is the only thing that makes the reported number mean anything.

The run below is the wall coming down. Prompt v7 scored 38 of 40 on dev and 29 of 40 on held out. The 11 held out failures were real: eight tickets with no order number, three with two orders, categories the dev set had under sampled. The engineer did the fast thing at line 4: pasted the eleven failing tickets into the prompt as examples. v8 scored 39 of 40 on held out. Report: 97.5%. Ship.

That 97.5% measures memory, not skill. The held out set has been seen by the prompt, so the next real ticket with two orders is no more likely to pass than it was with v7. The right move was slower: the eleven failures name two under sampled categories; go back to the trace store, pull fresh cases from those two categories into the dev set, fix the prompt against them, and run held out once more.

With forty cases, one case is 2.5 points. A move from 34 to 36 is two cases, and lesson 5.4 showed that two cases can flip on the sampler alone. So report counts, not just percentages: "36 of 40, up from 34, both at five runs each" says more than "90%". Report by category too, because 36 of 40 with all four failures in one category is a different product from 36 of 40 spread evenly.

Refresh on a schedule. Every month, retire the held out cases that have been stable longest into the dev set and replace them with fresh production traces, labeled the same way. The held out set stays small, stays real, and stays unseen.

The question is "how big was your eval set and where did the cases come from". The answer has four parts and takes twenty seconds. "Eighty cases, forty dev and forty held out. Sampled by failure category from last month's production traces, eight per category, a few synthetic ones marked as such. Labeled pass or fail with a reason by the support lead. Held out is run only for the report and refreshed monthly." Then stop. The follow up will be about one category, and you will have a story for it.