Design an agent that reviews pull requests

The interviewer says: "Design an agent that reviews pull requests." You have twenty minutes and a whiteboard. The weak answer starts with the model. The strong answer starts with five headings, written on the board before you say anything else: scope, tools, stop conditions, evals, cost. Everything you say after that goes under one of them.

This lesson is that page. The trace at the end is one planned run through the design, the way you would draw it in the last five minutes.

Input: one pull request, its diff, the files it touches, the description, and the CI result. Output: at most five comments, each tied to a file and a line, plus one line at the top saying what was checked. The agent never approves and never merges. A person does that.

Out of scope: style and formatting, because a linter already does it for free. Rewriting the change. Reviewing a diff over 800 changed lines: the agent posts one comment asking for a split and stops, because large diffs are where a reviewer, human or model, misses the most. Google reported in 2023 that about 7.5 percent of its code review comments were being resolved by model suggested edits. That is the scale to aim at: useful comments on a slice of the work, not a replacement for the reviewer.

Four tools. read_diff(pr) returns the change. read_file(path, lines) returns a window of a file at the base commit. search_repo(q) finds callers and definitions. post_comment(path, line, body) writes a comment. There is no tool that pushes, approves, merges or edits the branch. The tool list is the permission list: if the agent cannot call it, prompt injection in a commit message cannot make it happen either.

Read tools are cheap and get a count cap. post_comment is the only write, and it is capped at five per run and checked in code: the path must be in the diff and the body must be under 600 characters.

The loop ends when the first of these is true: the review is posted, twelve tool calls have been made, 60,000 input tokens have been spent, three minutes have passed, or the same tool call repeats with the same arguments. Each one is a line of code around the loop, not a sentence in the prompt. A prompt that says "be efficient" is not a stop condition.

When a cap fires before a review is posted, the agent posts nothing and logs why. A half review is worse than no review, because the reviewer assumes the rest was checked.

Evals: forty past pull requests where a person found a real bug, labeled by hand with the file and line. Two numbers. Of the forty bugs, how many did the agent comment on (recall). Of all the comments the agent posted, how many did the reviewer resolve rather than dismiss (a precision proxy from the live data). A judge for "is this comment useful" is calibrated against the same forty labels before it is trusted, as in Part 5.

Cost: three model calls per review, about 40,000 input tokens and 4,000 output. At the mid tier price on the Anthropic pricing page in October 2026, 2 dollars per million input tokens and 10 per million output, that is about 12 cents a review. Two hundred pull requests a week is about 24 dollars. Say the number out loud. Interviewers remember candidates who priced the thing.

In the room, go heading by heading and give one concrete item under each. Then draw the planned run. The whole answer fits on one page, and that is the point.