Retrieval and memory: context is a budget (2023)

A language model knows what was in its training data up to a cut off date, and nothing about your database, your tickets or your last conversation with it. And whatever you want it to use has to fit in the context window, which is a fixed number of tokens with a cost per token on every call. Both problems have the same answer: decide what goes in the window, and do it with code.

The idea is older than the chat era. In 2020 Patrick Lewis and colleagues at Facebook AI published Retrieval-Augmented Generation, RAG. Embed the question, find the nearest passages in a vector index of Wikipedia, put them in front of the generator. They trained the retriever and the generator together. By 2023 nobody trained the two together. You took a chat model, embedded your documents with an embedding model, stored the vectors, and at question time pulled the top few chunks into the prompt. Vector databases became a product category.

The results were good on the first demo and bad on the first real query. The chunk boundaries cut a table in half. The top five chunks all came from the same stale document. The question used a word the documents did not. Every one of those is a retrieval failure, not a model failure, and the fix is in the index, the chunking, the query rewriting or the ranking. Lesson 3.3 of this course is one of those runs.

In May 2023 Anthropic announced a 100,000 token window for Claude, enough for a novel, and the other providers followed. In July 2023 Nelson Liu and colleagues at Stanford published Lost in the Middle. Give a model twenty retrieved documents with the answer in one of them. Accuracy is high when the answer is first or last and drops sharply when it is in the middle. For some models, accuracy with the answer in the middle was below the same model's accuracy with no documents at all. A big window is not a big memory. It is a big budget, and the model spends attention unevenly across it.

Memory across conversations is retrieval with a write path. In April 2023 Joon Sung Park and colleagues at Stanford built Generative Agents: 25 characters in a small simulated town, each with a memory stream of everything it had seen, a retrieval step that scored memories by recency, importance and relevance, and a reflection step that wrote summaries back into the stream. In October 2023 MemGPT proposed treating the context window like RAM and an external store like disk, with the model itself issuing calls to page information in and out. Both are the same idea: the window is a cache, and something has to decide what is in it.

Treat context as a budget with line items. The system prompt costs this much. The tool definitions cost that much. The retrieved chunks get the rest, and they are ranked, not dumped. Log what went into the window on every run, so when the answer is wrong you can see whether the right fact was there at all. Most retrieval bugs are visible in that log in under a minute. Part 3 of this course is the interview round on exactly this.

Say that the window is a budget, cite the position effect, and say that the log of what went in is the first thing you read. Then name the next step. A model that can call tools, retrieve what it needs and write back to memory has everything it needs to run on its own: look, act, look again. In 2023 a lot of people wired that loop up and let it go. That is lesson 0.10.