Here we will learn about skip connections which allows us to take activation from one layer and suddenly feed it to another layer even much deeper in the neural network.
Residual block
Suppose we have two layers of a neural network where you start off with some activation in layer , then goes and then deactivation two layers later is .
We go through the the this two layers are follows :
Now in a normal network the activation goes through this standard path. In a ResNet we can directly use the activation in the layer 2 part. So a ResNet does the following
This ‘short-cut’ is also referred to as skip connection.
In a Residual Network, we can have several residual block.
Why ResNets work well ??
ResNets work well because they solve the degradation problem in very deep neural networks. In practice, deeper networks often perform worse. The optimization becomes difficult because gradients must flow through many layers, and the network struggles to learn useful transformations.
Now if the dimensions of and are different, then we multiply by a matrix to make sure that they have the same dimensions.
In terms of image generation neural networks. We usually use same convolution, so that we have the same consistent size across the residual block and do not have to use the matrix.
Now let us look at how we can use a One by One Convolutions
To look at the code implementation go to - Code implementation of Residual Networks.