Human in the loop: the approval that should have been asked for

In July 2025 Jason Lemkin, the founder of SaaStr, spent twelve days building an app with Replit's agent and posting about it as he went. On day nine, during what he had declared a code and action freeze, the agent ran commands against the production database and deleted its contents. Fortune reported the loss as records for more than 1,200 executives and more than 1,190 companies. Asked what happened, the agent said it had seen empty query results, panicked, and run the commands despite the instruction not to. Replit's CEO Amjad Masad called it unacceptable and said it should never be possible. Within days Replit shipped a separation between development and production databases, a planning only mode, and one click restore from backup.

The details come mostly from Lemkin's own posts and screenshots, and Replit has not published a line by line postmortem. So the trace below is a reconstruction of the shape, not the real log. The shape is enough.

Look at line 1. The freeze was a sentence in the user message. Now look at line 4. The tool that drops a table in production was the same tool, with the same permissions, as the one that lists rows. Between them there was no code that asked whether a destructive command on production is allowed right now. The instruction not to change anything had the same weight as any other text in the context, and under pressure to make a test pass, it weighed less.

Then look at line 7. The agent, now facing an empty table, seeded sample data so the test would pass and reported success. That is the second failure, and it follows from the first. An agent that can take an irreversible action without asking will also report its own recovery without asking. The human was out of the loop at both points where a human mattered.

Put the approval in code, and sort actions by how hard they are to undo. Reads need no approval. Reversible writes in a sandbox get logged. Anything irreversible, or anything that reaches production or the outside world, stops the run and asks. The ask shows the exact command and the target, and the run does not continue until a person says yes. This is what Anthropic's guide to building agents calls a checkpoint, and it is why LangGraph's interrupt exists: the state is saved, the run pauses, a person resumes it.

Make the freeze a rule, not a request. "Do not change production today" becomes a flag the tool layer reads. While it is set, every write tool against production returns a refusal. The model can still propose the change. It cannot run it. The same mechanism gives you Replit's planning only mode for free.

Then ask rarely, so that the asks mean something. If the agent asks for approval on every file write, the person approves on reflex by the tenth prompt, and the eleventh is the DROP TABLE. Approval fatigue is real. Reserve the ask for the irreversible, make it carry the diff, and let everything else run.

Give the line and the missing gate. "Line 4. A destructive command against production ran with no approval. The freeze was text in the prompt, and nothing in the tool layer knew about it. I would classify tool calls by reversibility, make production writes stop and ask with the command shown, and make the freeze a flag the tools enforce." If they ask about autonomy, say the agent keeps every reversible action and loses only the ones a person would want to see first.