A broken agent · free, no account

The judge that liked long answers

A logistics company's HR policy assistant is tested every night by a judge that compares its answers with HR's reference answers. Tap the line where it went wrong, write the fix, and a real model runs it on inputs you have not seen. About ten minutes. Your progress is kept, with or without an account.

Behind this run

Every run

One is today's. The rest are open too. The set grows by one a week.

1The double charge A support agent reads billing tickets and replies to customers. One prompt does everything.
2The release note with the wrong number An agent turns the weekly metrics table into a product update and posts it to the company channel.
3The tool called with the wrong argument A shop assistant agent can look up orders with one tool. A customer asks where their order is.
4The fraud report sent to tech support A bank's support inbox has one router in front of four teams. It reads each message and picks the queue.
5The plan that never stops searching A travel agent plans a booking first, then runs the plan with a flight search tool and a booking tool.
6The booking that failed and was confirmed anyway A clinic's scheduling agent books appointments through an API, then texts the patient a reply.
7The refund that came out a hundred times too small A shop's refund agent calls one tool that sends money back to the customer's card.
8The quote with a made-up trucking price A freight forwarder runs three price checks in parallel, ocean freight, customs and trucking, then one step merges them into a quote for the client.
9The leave type code could not read An HR agent handles leave requests in three steps: extract the dates, classify the leave type, then code routes the request by type.
10The opening hours that cost a fortune A law firm's client portal sends every question through a router that picks a small, cheap model or a large, expensive one.
11The migration declared complete with two tables missing A school district's ops agent moves the student records database to a new cluster. A monitor step decides whether the goal is met.
12The refund worked out in one jump A SaaS billing agent works out prorated refunds when a customer cancels, and code sends the number to the payment system.
13The email that gave orders An inbox assistant for a small design studio summarizes every new email. It can also forward and delete mail.
14The read-only agent that deleted a record An HR assistant answers questions about staff records. A permission checker runs before every tool call it makes.
15The 2,400 dollar refund nobody approved A furniture shop's support agent can issue refunds and store credit. An approval gate decides when a human must sign off first.
16The memory that saved the wrong things After each call, an internet provider's agent writes facts to the customer's long-term profile so the next call starts informed.
17The order that forgot it was packed A warehouse agent tracks each order through picking, packing and shipping, with events arriving from scanners all day.
18The answer from the wrong version A docs assistant for Ledgerly, an invoicing API, answers developer questions from retrieved documentation chunks.
19The policy rewritten by one angry message A bike-share service runs a feedback learner each week. It reads user feedback and proposes changes to policies and app flows.
20The patient's number in the team channel A clinic's phone agent summarizes each call and posts the summary to a shared front-desk channel that 30 staff can read.
21The meeting booked on the wrong day A law firm's scheduling assistant books meetings in the firm's calendar with one tool.
22Two agents agreed on the wrong number A furniture maker's procurement team runs two research agents and a verifier to answer supplier questions before a purchase plan is saved.
23The latency that lost its unit At a concert ticketing site, a monitoring agent reads the dashboards and sends a message to a paging agent that decides whether to wake the on-call engineer.
24The VIP's coffee before the gas leak A property management company runs a triage agent that orders the morning's facilities tickets for its one on-site technician.
25The eval that missed the made-up date An online eyewear shop scores every support reply with an LLM judge, and the dashboard said 96% of replies were good.
26The judge that liked long answers A logistics company's HR policy assistant is tested every night by a judge that compares its answers with HR's reference answers.
27The failures nobody could count The team behind a rail and hotel booking agent is turning last month's failed conversations into a held-out test set, and a labeling prompt tags each failure.
28Five retries of a request that could never work A marketplace's payouts agent sends money to sellers through a payments API, and a recovery policy decides what to do when a call fails.
29The agent that kept searching A school admissions helpdesk runs a ReAct agent: at each step it reads what it has found so far and picks the next action, with a budget of 10 steps.
30The currency passed in words A bank's travel money assistant converts amounts between currencies with one tool.