Tools and function calling: the model reaches outside (2023)

In October 2022 Shunyu Yao and colleagues at Princeton and Google published ReAct. The prompt showed the model a format: Thought, then Action, then Observation, repeated. Actions were plain text, such as search[Apple Remote]. A loop outside the model parsed the action, ran it against Wikipedia, and pasted the result in as the observation. On fact checking it beat chain of thought alone, and on question answering the two combined beat either. The paper's point was that a model that can look things up makes fewer things up.

In February 2023 Meta published Toolformer, which fine tuned a model to insert calls like Calculator(400 / 1400) into its own text, and taught it when a call helped by checking whether the result lowered the loss on the next tokens. It is the paper that showed a model could learn when to use a tool, not just how.

Both had the same weakness. The action was free text. The model would write search[Apple remote]] with a stray bracket, or call a tool that did not exist, or put the arguments in the wrong order. Every team wrote its own parser and its own retry logic, and every parser broke.

OpenAI shipped ChatGPT plugins in March 2023, letting ChatGPT call outside services from a description of their API. Then on 13 June 2023 it released function calling in the API for gpt-4-0613 and gpt-3.5-turbo-0613. You send a list of functions, each with a name, a description and a JSON Schema for its arguments. The model, fine tuned for this, replies either with text or with a structured call: the function name and a JSON object of arguments that matches the schema. Your code runs the function and sends the result back as a message with its own role. The model then writes the answer, or calls again.

Anthropic shipped the same shape for Claude, generally available on 30 May 2024, and Google did the same for Gemini. By mid 2024 the function calling message format was the one thing every provider agreed on, and that agreement is what made the protocols in lesson 0.11 possible.

Three things. The call became structured, so a parser could not break it. The model was trained to decide whether to call, which tool, and with what arguments, from the schema alone. And the result came back as its own message with its own role, so the model knew it was a tool result and not something the user said. The trace below shows the whole loop for one weather question. Note the shape: the tool line is not the model's words, and the final answer is written from the tool result, not from memory.

The schema is a prompt. A function named get_data with an argument called x gets called wrong. A function named get_customer_orders with customer_id described as the numeric id from the account page, not the email, gets called right. The model reads the description on every turn, so write it for the model. Also: the model will call a tool that does not fit if it is the only tool it has. Give it a way to say none of these. Lesson 1.2 of this course is a run where the argument was wrong.

Say that function calling moved the contract from the parser to the model and the schema. Then name what it opened. Once the model can call tools, the obvious tool is search over your own documents. The model's weights are frozen at training time and know nothing about your data, and its context window is a fixed budget. How you fill that budget became the next problem. That is lesson 0.9.