In my previous post we left the story with the Perceptron and its limitations, one of them being the lack of a suitable algorithm to deal with the multiple layers of neural networks. By 1986 a candidate for such an algorithm had appeared, with the somewhat fancy name of backpropagation (for a full explanation of this and other concepts, see this post). I always read about how backpropagation laid the foundation of the deep learning revolution in 1986. But if that is the case, then why didn't deep learning explode already in 1986 if it all was so wonderful? The short answer is that, for networks with more than two or three hidden layers, backpropagation didn't work very well.
To understand why, recall what backpropagation does (click here for a more detailed explanation of how neural networks in general function). We try to find in which direction (up or down) our weights and biases need to be tuned in order to decrease the loss function and improve the predictive power of our network. We perform this for each layer in the network, starting at the last layer (the layer before the output layer) and sequentially going back through each layer until we are at the input layer. It was well recognized that adding more layers to neural networks would improve their performance and allow them to recognize and predict more and more hidden features within a dataset. But as the networks grew in size and more layers got added, the learning became more and more difficult. The learning signals (the numbers indicating the direction in which to tune the weights) became weaker and weaker as the backpropagation algorithm moved through the different layers. Researchers dubbed it the vanishing gradient problem, and it drove them to despair.
We have already seen several of these episodes throughout AI's brief but sometimes brutal history. Periods where expectations and hopes were sky-high, only for these hopes and dreams to run into practical realities or constraints. What followed was a time where funding dried up, researchers lost faith, and AI largely disappeared from serious scientific conversation. The dreaded "AI winter", and it was back with a vengeance.
It would take more than a decade and several developments to tame the vanishing gradient problem. Many of these improvements were highly technical and mathematical, but I want to highlight three which I found to be fascinating in their own regard.
- Initializing the network with weight and biases. Did you ever wonder how we get to our initial set of parameters (the weights and biases)? I always thought you would just randomly pick them out of a given statistical distribution. But if you do that for a deep neural network, you run into other problems. During the feed forward pass the signals (remember these are the weights multiplied by the value of the input layer) tend to grow either too large (researchers coined the term exploding signals) or too small (vanishing signals). Because the signal either exploded or vanished on the very first forward pass, the gradients computed during backpropagation were completely broken from the first millisecond the network became live. It was dead before it completed its first pass. The fix turned out to be relatively simple. It is based on fancy statistics, but roughly speaking you had to ensure the initial random weights were taken from a random distribution but with a fixed variance, which was related to the size of the input and output layer. They were selected to be "just right" to avoid either exploding or vanishing signals.
- Skip connections. I found this idea quite strange to begin with: to avoid the issues with layer after layer multiplying signals or gradients, why don't we just ... skip some layers? It seems a silly idea, and yet it turned out to work just fine in practice. A skip connection exists in parallel to the normal connection: the normal connection goes back to the previous layer, whereas the skip connection goes back 2 or 3 layers; it skips over some of the layers in between. Where the two connections meet again, the signals are just added together, not multiplied. If the normal route encountered the vanishing gradient problem and got very small, the signal can be "restored" by adding the skip connection signal back to it. In a deep layered network these skip connections jump over layers each time. The outcome is a much more stable network which is less affected by the vanishing gradient. I know, it sounds crazy and it certainly baffles me, but it just shows that designing these networks is still very much a case of trial and error, and crazy ideas sometimes just get things done.
- The ReLU activation function. This is my favourite fix. Every neuron in every neural network has an activation function. It's the one place in the network where something different from adding or multiplying numbers happens. It's a crucial piece in the machinery to allow neural networks non-linear complex behaviour (because the real world is messy and not linear). Researchers had come up with all kinds of beautiful activation functions for very good reasons (if you're interested in learning more, google the sigmoid or tanh functions). The ReLU should not have worked ... it just takes whatever value the neuron obtained, and if it's a positive number keeps it, and if it's a negative number then it just ignores it and returns a zero (technically it takes the maximum value of zero and the input, therefore always returning at least a zero). No fancy mathematical functions, no smooth curves. I always compare it to a guillotine which cuts away all negative numbers. But it worked, neural networks with the ReLU performed much better than the exact same networks with classical activation functions. It does not always have to be complicated.
The above, together with many other improvements, finally allowed neural networks with tens or hundreds of layers to become effective. The most famous demonstration happened in September 2012. For several years, an annual competition called the ImageNet challenge was organised, in which neural networks competed to classify images. In 2012 a deep neural network with 8 layers and 60 million parameters, called AlexNet, won the competition. But it did not just win, it blew the competition out of the water with its performance. Improvements were usually measured in fractions of a percent better error rates. AlexNet managed a full 10 percent improvement compared to the next competitor. The AI Winter was over. Deep neural networks were finally a reality.
One of the key figures in the above story is Geoffrey Hinton, one of the so-called fathers of AI. He was instrumental in several key developments mentioned here: co-author of the seminal 1986 backpropagation paper, fighting against the vanishing gradient and motivating his researchers to use the ReLU activation function in building AlexNet. He was one of those researchers who kept going in the face of budget cuts, the common belief that deep neural networks would just never work, the many disappointments. It's a nice illustration that personalities do matter, and the story of AI is ripe with them. Hinton's persistence and refusal to give up culminated in him winning the 2024 Nobel Prize in Physics, together with John Hopfield. They were nominated "for foundational discoveries and inventions that enable machine learning with artificial neural networks."
AlexNet was significant for another reason as well. By now the backpropagation algorithm, the ReLU activation function and the many other improvements had shown the way forward. Deep neural networks could be made to work. The calculations were doable. But the amount of computations proved to be a formidable challenge, and classical computer chips (CPUs) were just not up to the task. Something else was needed. And it needed to be blazingly fast.
The story continues in my next post, where we turn our attention to the advancements in hardware which made today's AI revolution a reality. As always please drop me a note if you find this interesting, or if you want me to cover other topics in my posts. I would love to hear from you!