Sequence models and attention (2014 to 2017)

A sentence has no fixed length, and the words that matter for the one you are reading can be far behind it. The standard tool was the recurrent network: read one token, update a hidden state, read the next. The LSTM, from Sepp Hochreiter and Jürgen Schmidhuber in 1997, added gates so the state could carry information across long gaps without being overwritten at each step.

In 2014 Ilya Sutskever, Oriol Vinyals and Quoc Le at Google trained two LSTMs end to end for translation. One read the English sentence into a single vector. The other wrote the French sentence out from that vector. On the WMT 14 English to French task an ensemble of their models scored 34.8 BLEU, against 33.3 for the phrase based statistical system that was the standard. One odd trick helped a lot: reversing the order of the source words, which put the start of each sentence closer to its translation.

Squeezing a whole sentence into one fixed vector works for short sentences and fails for long ones. The same year Dzmitry Bahdanau, Kyunghyun Cho and Yoshua Bengio published a fix. Instead of one vector, keep every encoder state. When the decoder writes each word, let it compute a weight over all the source positions and take a weighted sum. They called it alignment. Everyone else called it attention. The model learned on its own which source word each output word should look at, and the alignment maps in the paper look like the ones a linguist would draw.

Attention sat on top of recurrence for three years. The recurrence was still the problem. To process token 50 you must first process tokens 1 to 49, in order, so one sequence cannot be spread across a GPU. Training was slow, and long range information still had to pass through a chain of hidden states, one step at a time.

In June 2017 eight people at Google published Attention Is All You Need. The transformer keeps the attention and drops the recurrent network. Every token attends to every other token in the same layer, in one matrix multiplication. Position is added as a separate signal because the model no longer reads in order. Stack the layers, put a feed forward block in each, and that is the whole architecture.

The numbers made the argument. The big model scored 28.4 BLEU on English to German and 41.8 on English to French, both the best published at the time, after 3.5 days on eight P100 GPUs. The base model trained in twelve hours. The paper's own table shows that at a fraction of the training cost of the previous best systems.

Say this plainly in an interview. Recurrence is sequential by construction, so it fights the hardware. Attention is a matrix operation, which is the hardware's favourite shape. And attention gives every token a direct path to every other token, where recurrence gives a path of length n through a state that leaks. The cost is that attention is quadratic in sequence length. In 2017 nobody cared, because sentences are short. By 2023, when people wanted contexts of a hundred thousand tokens, everybody cared. That is lesson 0.9.

A transformer trains fast, scales with GPUs, and has no built in assumption about the task. Translation was just the first job. The question in 2018 was what happens if you train one on all the text you can find, with no task at all, and ask it only to predict the next word. That is lesson 0.5.