Notes on Weight Initialization for Deep Neural Networks

Now, if we scale the weights of the matrix and divide them by , we will be picking elements of from a normal distribution with mean and variance = or (see the next section for a proof). This will in turn give us a distribution of in which each element has 0 mean and std = 1, thus allowing us to repeat the product as many times as we want (or in other words, make our network deeper). Dividing weight matrix by (num_inputs = 512 in the running example), known as Xavier Initialization, helps in ensuring that the output of each of the multiplications is well-behaved, thus ensuring that the sequence of multiplications yields a reasonable output at each step.

Source: madaan.github.io