Clojure from Scratch to GPU: A Simple Neural Network Training API

The root of this issue is in temporary working memory that CLBlast creates during matrix multiplication. If you just launch many multiplications of matrices of the same sizes, or smaller matrices, these temporary buffers get created and destroyed, and everything works well. Reductions are bad for GPU performance, but in this case matrix multiplications are much more demanding, so this should not take a big impact.

Source: dragan.rocks