Reflection: the release note with the wrong number

An agent turns the weekly metrics table into a product update and posts it in the company channel. This week's source line: "Checkout conversion: 3.1% -> 3.4% (+0.3 pts)." The prompt asked for an upbeat update. The draft said: "Big week: checkout conversion is up 30%!" The agent posted it to #announcements. The run ended with status success. By lunch the head of growth was answering questions about a number that did not exist.

The slip itself is ordinary. Models turn 0.3 points into 30 percent, 1,240 to 1,302 into 50 percent, and "roughly flat" into "record growth" when told to be upbeat. This is the same run as the live challenge for this lesson, and the trace below is the broken one you will fix there.

Line 2 is where it went wrong, and it does not look wrong. The prompt asked for one thing, an upbeat draft, and got it. What it did not ask for was anyone to check the draft against the source. The run went from draft to post_message with no step between them, and post_message is irreversible: a channel full of people saw it.

The deeper point is that a single call has no second opinion. The model that wrote 30% was, at that moment, confident. Asking the same model in the same call to double check its numbers rarely helps, because it is still one pass with one context. What helps is a separate call with a different job and the source in front of it.

Add a critic step between draft and post. The critic receives two things, the source table and the draft, and does one job: check every claim in the draft against the source. It returns JSON, not prose: a verdict of approve or revise, and a list of issues that each name the claim, the source value and the fix. Code reads the verdict. On revise, the issues go back to the drafter and it tries again. On approve, the post goes out. Nothing is posted until the critic approves, and the critic has no post tool.

Three rules make the critic worth its cost. It checks facts, not tone, because tone is what the first prompt already optimized for. It returns a structured verdict, because a paragraph of feedback cannot be branched on. And it does not rewrite the draft, because a critic that rewrites is a second drafter with the same blind spots. Anthropic calls this shape evaluator-optimizer in its 2024 guide to building agents. The research behind it goes back to March 2023: Self-Refine reported about 20 points of average gain across seven tasks on GPT-3.5 and GPT-4 from a draft, critique, revise loop, and Reflexion fed an agent's own written critique back into its next attempt.

Be honest about the limit. A critic from the same model family catches slips like a wrong number, where the source is in the context and the comparison is mechanical. It is weaker on errors the model cannot see, such as a claim that is plausible and wrong. For those you need a tool, a lookup or a person, not another model call. And the loop costs a call per round, so cap it at two or three rounds and spend reflection where a wrong output is expensive or public.

Say the line, the missing step and the shape of its output. "Line 2. The prompt asked for an upbeat draft and nothing checked the draft against the source before an irreversible post. I would add a critic call that gets the source and the draft, checks every number, and returns a JSON verdict the code can branch on. The critic does not post and does not rewrite." If they ask about latency, say you keep the critic on anything with numbers or commitments and skip it on low stakes drafts.