The flaky eval: the test that passes on Tuesdays

Case 17 in the support suite passed on Tuesday 13 October and failed on Monday 19 October. Nothing in the repository changed between the two runs. The team spent a morning diffing prompts. The trace below shows the cause in one line: the system prompt says "Today is {date}. Urgent means due within 3 days." The ticket says "before Friday". On Tuesday, Friday is three days away, so the model said high and the case passed. On Monday, Friday is four days away, so the model said normal and the expected label, written on a Tuesday, was wrong.

The eval was measuring the calendar. It would pass every Tuesday, Wednesday and Thursday and fail every Monday, forever.

Flaky tests are old news. Google reported in 2016 that 1.5% of its test runs gave a flaky result and almost 16% of its tests showed some flakiness. Evals of model calls add four new causes on top of the usual ones.

The sampler. Any temperature above zero is a coin flip by design. Temperature zero is better but not deterministic: Thinking Machines showed in September 2025 that 1,000 completions at temperature zero from one open model gave 80 distinct outputs, because the kernels on the server change their arithmetic with batch size. The judge. It is a model too, with the same sampler, so a flaky judge can fail a stable output. The clock and the world. Dates in the prompt, live tool calls, search results, exchange rates. The vendor. A model name that is an alias moves underneath you when the provider updates it.

The tau-bench paper from 2024 put a number on how bad this gets for agents: the best agent they tested passed fewer than half the tasks once, and when they required the same task to pass eight runs in a row (pass^8), it was under 25% in the retail domain. One green run is a weak claim.

Pin what you can. The date is part of the case input, not read from the clock: case 17 carries now = 2026-10-13 and the expected label is right forever. Tool results are recorded once and replayed, so the eval never calls the live API. The model is a dated version, not an alias, and the eval log records which one ran. The judge runs at temperature zero on a pinned model and its rubric is versioned.

Count what you cannot pin. Run every case more than once. Report the pass rate per case, not a single boolean, and for the cases that gate a deploy require all runs to pass, which is pass^k. A case at 3 of 5 is a flaky case, and a flaky case is a bug report: either the prompt is sitting on a decision boundary, or the case is ambiguous, or the check is too tight. Quarantine it from the gate, but do not delete it, and give someone the ticket.

Alert on the trend, not the run. One failure on a 40 case suite is a 2.5 point move. Three runs in a row below last week's rate is a signal. One run is noise until proven otherwise.

Some flakiness is the product telling the truth. If a case passes 3 of 5 times with everything pinned, the agent really does get that case wrong two times in five, and customers see the same odds. Do not fix the eval. Fix the agent, and keep the case at pass^5 so the fix has to hold.

"Flaky evals come from the sampler, the judge, the clock and the vendor. I pin the model version, freeze dates and tool results inside the case, run the judge at temperature zero, and run every case five times. The deploy gate is pass^k on the cases that matter. A case that flakes with everything pinned is a real defect rate, and that goes to the backlog, not to the delete key."