Deep Learning in Clojure from Scratch to GPU: Initializing Weights

Here’s simple example, an one layer network we used in part 1, Representing Layers and Connections:

Take a look at the formula for the linear transformations that we defined in that article:

Each \(h_i\) is a dot product of the respective row of \(W\) and the input. Until now, we have set the initial weights and inputs to be in the range \([0, 1]\), as in the following example that you have seen many times by now. Although the saturation will contain the inputs to the next layer to the \([-1,1]\) range, it would make the learning difficult, since the saturated functions would have problems propagating the gradients backwards.

Source: dragan.rocks