Backpropagation and the hidden layer (1986)

On 9 October 1986 Nature printed a four page paper by David Rumelhart, Geoffrey Hinton and Ronald Williams called Learning representations by back-propagating errors. It answered the question Perceptrons had left open. Put a layer of hidden units between input and output. Use a smooth activation instead of a hard threshold. Then the error at the output can be pushed backwards through the network with the chain rule. Every weight gets a number: how much the final error would change if that weight moved a little. Then you move it.

XOR falls to a network with two hidden units. The paper showed a network that learned to detect mirror symmetry in a row of inputs, and one that learned family relationships from a list of facts. That second network grew hidden units that stood for things like nationality and generation without being told those concepts existed. This is the demo that matters. The hidden layer learned a representation nobody designed.

Be honest about this one in an interview. Reverse mode differentiation was published by Seppo Linnainmaa in 1970. Paul Werbos described using it to train neural networks in his 1974 Harvard thesis. The 1986 paper is the one that showed the method on neural networks with clear experiments, in the most widely read journal in science, with the two volume Parallel Distributed Processing books behind it. It is the paper that made people use the idea, not the first paper to have it.

Yann LeCun's group at Bell Labs applied backpropagation to handwritten zip code digits from the US Postal Service in 1989. The network shared weights across positions so that one small detector scanned the whole image, which is the start of the convolutional network. It trained on 7,291 digits and was tested on 2,007, with about 5 percent error on the test set. The 1998 follow up, LeNet-5, was deployed commercially and read several million cheques a day.

Three things held it back. Networks with more than a couple of hidden layers were hard to train, because the error signal shrinks as it passes back through each layer. Sepp Hochreiter described this vanishing gradient in his 1991 thesis. Computers were slow, so training was measured in days for tiny networks. And in the 1990s support vector machines and other kernel methods gave better results on most benchmarks with a cleaner theory behind them. By 2000 a neural network paper was a hard sell at a major conference.

The method was right. What it needed was far more data and compute than anyone had. The people who kept going, Hinton in Toronto, LeCun in New York, Yoshua Bengio in Montreal, spent the 2000s on tricks for training deeper networks and on arguing that scale would work. The field did not believe them until a leaderboard said so.

The idea to carry forward is that an algorithm is one of three ingredients. Backpropagation was ready in 1986. The data and the compute were not. When a method stalls, ask which ingredient it is waiting on before you blame the method. The breakthrough came when a graduate student put the old method on a new kind of chip, with a million labeled images. That is lesson 0.3.