Knowledge retrieval: the answer grounded in the wrong document

A help agent answers policy questions from the company's documents. It searches an index, takes the top chunks, writes an answer, and cites the chunk it used. That last part was supposed to make it safe. A customer asked how long they had to return a laptop. The agent answered 30 days and cited returns-policy-2023.pdf, page 2. The current policy, in force since January 2025, says 14 days for laptops and tablets. The customer came back on day 21 with the chat transcript.

The answer was grounded. It was cited. It was wrong. Look at line 3 of the trace: the 2023 chunk came back first with a score of 0.81, the 2025 chunk second at 0.79. Both were in the index. Nothing marked one as current.

Retrieval is a pipeline and each stage can fail on its own. Chunking cuts documents into pieces of a few hundred tokens. If a chunk loses its heading, it loses the fact that it belongs to the 2023 policy. Ranking scores chunks by similarity to the question: by embedding, by keyword (BM25), or both. Similarity does not know about dates. A superseded page that uses the same words as the question scores as well as the current one, and often better, because old pages are longer and repeat themselves. Citation then points at whatever won. A citation proves the model did not invent the text. It does not prove the text is the right text.

Anthropic measured one piece of this in 2024. Adding a short context line to each chunk before indexing, with embeddings and BM25 together, cut the top-20 retrieval failure rate by 49 percent, and adding a reranker on top cut it by 67 percent. That is the chunking stage and the ranking stage, each improved on its own. Neither would have fixed this run, because the 2023 page is a good match for the question. This run needed a filter.

The field splits on how much of this to do with embeddings and how much with plain metadata. My view: anything with a date, a version or an owner is metadata and belongs in a filter, not in the vector.

Fix the index before the prompt. Every chunk carries its document id, the document version, an effective date, and a current flag. Ingestion sets current to false on the old version when the new one lands. The retriever filters on current=true by default, and a separate query lets a person ask about history on purpose. That is the one line fix for this run.

Then make chunks carry their context: the document title and section heading go at the top of every chunk, so a piece of the 2023 policy says so in its first line. Use hybrid ranking, keywords and embeddings together, and rerank the top 20 with a cross-encoder when latency allows. Take fewer chunks into the prompt, three to five, and put them before the question (Liu et al. 2023 for why position matters).

Then measure retrieval on its own. Keep a set of 40 to 50 questions with the chunk that should win, labeled by hand, and track recall at 5 after every index change. The generation step gets a different check: does the answer use the cited chunk and only the cited chunk. When the two checks disagree you know which stage to open.

"Line 3. The index held two versions of the policy and ranked by similarity, so the superseded one won and the citation made it look safe. I would add an effective date and a current flag to every chunk, filter before ranking, and keep a labeled set to measure recall." Then name the three stages, chunking, ranking, citation, and say which one this bug lives in. Interviewers for RAG roles want to hear that a citation is not a correctness check.