Instruction tuning and RLHF: the model that follows orders (2022)

In 2021 you could ask GPT-3 to explain the moon landing to a six year old and get back a list of more requests for explanations. The model was trained to continue text, and in the training data a request is usually followed by another request, not by an answer. OpenAI's own paper opens with that example. Users worked around it with prompts that looked like the start of a dialogue. That is a workaround, not a product.

The first piece was supervised fine tuning on instructions. Google's FLAN paper in late 2021 took a 137 billion parameter model and fine tuned it on more than 60 datasets rewritten as instructions. On tasks it had never seen, it beat zero shot GPT-3 on 20 of the 25 datasets tested. A model that has seen a few thousand examples of instruction then answer learns the format and applies it to new instructions.

The second piece was reinforcement learning from human preferences, which Paul Christiano and colleagues at OpenAI and DeepMind had published in 2017 for games and simulated robots. Instead of writing a reward function, show a person two short clips and ask which is better. Train a model to predict that choice. Use the predicted reward to train the agent. In 2020 OpenAI applied the same loop to summarising Reddit posts and found that people preferred the summaries from a model trained this way over those from a much larger model fine tuned the ordinary way.

In March 2022 OpenAI published Training language models to follow instructions with human feedback, the InstructGPT paper. Three steps. Collect prompts from the API and have a team of about 40 contractors write good answers; fine tune on those. Have the contractors rank several model outputs per prompt; train a reward model on the rankings. Then optimise the language model against the reward model with PPO, with a penalty for drifting too far from the fine tuned model.

The headline result is the one interviewers ask about. Outputs from the 1.3 billion parameter InstructGPT were preferred by the labelers to outputs from the 175 billion parameter GPT-3, despite having more than 100 times fewer parameters. The model also made things up less often and was less toxic, with small losses on some public benchmarks that the paper calls the alignment tax.

Anthropic published its own version in April 2022, training a helpful and harmless assistant with the same loop, and in December 2022 added Constitutional AI, where a written set of principles replaces some of the human labels. Within a year the method was the industry standard.

On 30 November 2022 OpenAI put an InstructGPT style model behind a chat box and called it ChatGPT. The announcement describes it as a sibling of InstructGPT trained the same way with dialogue data added. Nothing in the model was new. The interface, and the fact that it did what you asked, were new to the public. What matters for this course is that the demand for people who could build on these models went from a niche to a job title within a year.

Say that RLHF changed what the model was trying to do, not what it knew. The 175 billion parameter model already contained the answers; the training made it give them. Then say what that unlocked. Once the model follows instructions, the instruction becomes the program. You can tell it to think in steps, to answer in JSON, to refuse certain requests, to follow examples. That turned prompting from a trick into engineering. That is lesson 0.7.