The take home, done right: a small agentic RAG system with a scored eval, in an evening

The take home says: build a question answering system over this corpus. Most candidates build the system. The ones who get the offer build the measurement. A submission with twenty labeled questions and a score of 14 out of 20 beats a submission with a clever agent and no number, because the reviewer can see what the second one does only by running it, and they will not.

Retrieval augmented generation is the 2020 idea from Lewis and others: fetch passages, then generate with them in the context. Agentic RAG adds a loop: look at what came back, decide whether it answers the question, and search again if not. The loop is worth building only if the eval shows the plain version failing on retrieval. So the eval comes first.

Hour one: corpus and questions. Pick a corpus you can read in twenty minutes, 30 to 60 pages of one project's docs. Write twenty questions by hand, each with the answer and the passage it comes from. Make five of them unanswerable from the corpus. Those five are the ones that separate a system from a demo.

Hour two: the plain pipeline. Chunk by heading, 300 to 500 tokens a chunk. Embed. Top five by cosine. One model call that answers with a citation to the chunk id, or says "not in the corpus". Save every run as a JSON file with the question, the chunks, the scores and the answer.

Hour three: the eval. Two code checks and one judge. Check one: is the labeled passage in the top five (retrieval hit). Check two: does the answer cite a chunk that contains the labeled passage, and does it say "not in the corpus" on the five unanswerable ones. The judge: a second model call scoring answer correctness against your label, 0 to 2, read against your own twenty labels before you trust it. Run it. Write the numbers down.

Hour four: make it agentic where the numbers say to. If retrieval hit is 13 of 20, the loop is: retrieve, ask the model whether the chunks contain the answer, if not rewrite the query once and retrieve again, then answer or refuse. Cap at two retrievals. Rerun the eval. The README gets a two row table: before and after.

The agentic part is one decision: do these chunks answer the question. That decision is a model call with a yes or no output, and the loop branches on it. Everything else is the plain pipeline. The cap is two retrievals, and the refusal path is a real path: when the second retrieval also fails the check, the system says it did not find the answer and shows the best chunk it had. A system that answers anyway is the broken trace below.

List the things you did not build and what they would buy. Anthropic's contextual retrieval post from 2024 measured a 49 percent drop in failed retrievals from adding chunk context before embedding and combining with BM25, and 67 percent with a reranker on top, on their top 20 benchmark. The lost in the middle paper from 2023 showed models reading the start and end of a long context better than the middle, which is why you keep top five, not top twenty. Naming these with numbers, and saying you skipped them to spend the hours on the eval, reads as judgment. Building them without an eval reads as a demo.

When they ask about the take home, lead with the table. "Twenty questions I labeled by hand, five unanswerable. Plain pipeline: retrieval hit 13 of 20, correct 11 of 20, two of the five unanswerable answered anyway. With one query rewrite and a refusal check: hit 17, correct 15, zero false answers. I skipped reranking; the post I read puts it at about a 67 percent cut in misses and I would try it next." That is ninety seconds and it contains every number they wanted.