Faster Parallel Python Without Python Multiprocessing

10x Faster Parallel Python Without Python Multiprocessing
Faster Python without restructuring your code
While Python’s multiprocessing library has been used successfully for a wide range of applications, in this blog post, we show that it falls short for several important classes of applications including numerical data processing, stateful computation, and computation with expensive initialization. Code for a toy image processing example using multiprocessing.The difference here is that Python multiprocessing uses pickle to serialize large objects when passing them between processes. The variability of the Python multiprocessing code comes from the variability of repeatedly loading the model from disk, which the other approaches don’t need to do.This example takes 5s with Ray, 126s with Python multiprocessing, and 64s with serial Python (on 48 physical cores).

Source: towardsdatascience.com