Adding layers to the middle of trained network without invalidating the weights

At this point, we can try creating a new bigger model that uses all the weights trained here (including those in the very last layer!) The important bit here is to initialize the layer so the weights are in the form of the identity function – this way when the output from the previous layer gets multiplied by the output of this layer we will get exactly the same result, and the weights of the final layer will still make sense. Now, the only other thing we need to do is add the weights from our previous model to our new model, except for our new (3rd) layer which will at first just leave things the same.

Source: svilentodorov.xyz