Data and GPUs: AlexNet (2012)

ImageNet was started by Fei-Fei Li, first at Princeton and then at Stanford, and described in a 2009 paper. Her argument was that the field had been working on algorithms while starving them of data. Her team used Amazon Mechanical Turk to label millions of images into categories taken from WordNet. The yearly contest built on it, ILSVRC, used about 1.2 million training images across 1,000 classes. The score was top-5 error: how often the right label was missing from the model's five best guesses.

In 2010 the winner had a top-5 error of 28.2 percent. In 2011 it was 25.8 percent. These were pipelines of hand built features such as SIFT and Fisher vectors feeding a linear classifier. Progress was a couple of points a year.

In 2012 a team from Toronto, Alex Krizhevsky, Ilya Sutskever and Geoffrey Hinton, entered a deep convolutional network. Top-5 error: 15.3 percent. Second place: 26.2 percent. The gap was bigger than the whole field's progress over the previous two years. The paper, ImageNet Classification with Deep Convolutional Neural Networks, came out at NeurIPS that December.

The network had eight learned layers, five convolutional and three fully connected, 60 million parameters and 650,000 neurons. It trained for five to six days on two NVIDIA GTX 580 GPUs, consumer graphics cards with 3 GB of memory each. The model was split across the two cards because it did not fit on one.

Not much, and that is the point. The convolution idea was LeCun's from 1989. Backpropagation was from 1986. The new pieces were engineering. ReLU activations instead of sigmoids, which the paper reports reached a target error six times faster on a smaller dataset. Dropout, which switches off half the hidden units at random during training so the network cannot rely on any one of them. Data augmentation by cropping and flipping images. And a GPU implementation of convolution that Krizhevsky wrote himself.

The lesson a working engineer takes from this is that the method had been waiting on data and compute for 23 years. When both arrived, the old method won by eleven points. Hinton's group had been saying this through the 2000s and nobody believed it until the leaderboard did.

Every leading entry in 2013 was a convolutional network. By 2015 a Microsoft network with 152 layers scored 3.57 percent top-5 error, below the 5.1 percent that Andrej Karpathy measured for himself on the same task. The GPU became the unit of research, and the people who could write for it became the scarce resource.

Say that AlexNet was a data and compute result, with a few training tricks, on top of a method from 1986. Then name the gap it left. Images are a fixed size grid. Text is a sequence of any length, with dependencies that reach far back. Convolution does not fit that shape. The question in 2013 was whether the same recipe, big network, big data, GPUs, would work for language. The answer came through sequence models and attention. That is lesson 0.4.