Pretraining: the language model (2018 to 2020)

In June 2018 OpenAI published a short report by Alec Radford and colleagues with a plain title: Improving Language Understanding by Generative Pre-Training. The model was a twelve layer transformer decoder with 117 million parameters. It was trained on over 7,000 unpublished books to do one thing: predict the next token. Then it was fine tuned, with a small extra layer, on each of twelve benchmark tasks. It set the best published score on nine of them. The paper is now called GPT-1.

Four months later Google published BERT, a transformer encoder trained to fill in masked words and to say whether two sentences follow each other. The large version had 340 million parameters. It took the top of the GLUE benchmark and became the default model for anything that classified text for the next three years.

Both papers made the same point. Train on raw text with no labels, where the data is nearly free. The model learns grammar, facts and some structure as a side effect of predicting words. Then a little labeled data on top does what used to need a big labeled dataset from scratch.

In February 2019 OpenAI announced GPT-2, 1.5 billion parameters, trained on web pages linked from Reddit. The title made a claim: Language Models are Unsupervised Multitask Learners. Give it a passage and the letters TL;DR and it summarises. Give it English text and a cue for French and it translates, badly. Nobody trained it to do either. OpenAI released the full model in stages over the rest of the year, out of worry about misuse, which was itself a signal of how the field saw it.

In January 2020 Jared Kaplan and colleagues at OpenAI published Scaling Laws for Neural Language Models. Loss falls as a smooth power law in three quantities, parameters, data and compute, across more than seven orders of magnitude. The curve did not bend. That paper is the reason the next model was a hundred times bigger.

In May 2020 GPT-3 arrived with 175 billion parameters, trained on about 300 billion tokens. The title was Language Models are Few-Shot Learners. The finding that changed the industry was in the prompt. Put a few examples of a task in the context, with no gradient update at all, and the model does the task. Translation, arithmetic, using a made up word in a sentence, unscrambling letters. Scores rose with model size on nearly every task, and the biggest model did things the smaller ones could not.

The paper is also honest about the failures. The model repeats itself over long passages, loses the thread, and states false things with full confidence. It was trained to continue text, so it continues. Ask it a question and it may answer, or it may write three more questions in the same style.

Pretraining turned the model from a thing you train into a thing you prompt. That is why there is now a job where you write text to a model and get software out. It also fixed the economics: one big training run, paid for once, then rented out by the token to everyone.

GPT-3 would do what you showed it, not what you told it. The gap between a text continuer and an assistant was the next problem, and it was solved not with more parameters but with people rating outputs. That is lesson 0.6.