Multiprocessing - shared memory with multidimensional numpy array

Question

I'm in a situation where I need to parallel process a very big numpy array (55x117x256x256). Trying to pass it around with the usual multiprocessing approach gives an AssertionError, which I understand to be because the array is too big to copy into each process. Because of this, I would like to try using shared memory with multiprocessing. (I'm open to other approaches, provided they aren't too complicated).

I've seen a few questions asking about the use of python multiprocessing's shared memory approach, e.g.

import numpy as np
import multiprocessing as mp

unsharedData = np.zeros((10,))
sharedData = mp.Array('d', unsharedData)

which seem to work fine. However, I haven't yet seen an example where this is done with a multidimensional array.

I've tried just sticking the multidimensional array into mp.Array which gives me TypeError: only size-1 arrays can be converted to Python scalars.

unsharedData2 = np.zeros((10,10))
sharedData2 = mp.Array('d', unsharedData2)## Gives TypeError

I can flatten the array, but I'd rather not if it can be avoided.

Is there some trick to get multiprocessing Array to handle multidimensional data?

Recent related question stackoverflow.com/q/50235377/901925

hpaulj
– hpaulj

2018-05-10 15:29:59 +00:00
Commented May 10, 2018 at 15:29 — hpaulj
– hpaulj, Commented May 10, 2018 at 15:29

dankal444 · Accepted Answer · 2021-05-03 16:39:55Z

1

You can use np.reshape((-1,)) or np.ravel instead of np.flatten to make a 1-dimensional view of your array, without unnecessary copying that flatten does:

import numpy as np
import multiprocessing as mp

unsharedData2 = np.zeros((10, 10))
ravel_copy = np.ravel(unsharedData2)
reshape_copy2 = unsharedData2.reshape((-1,))
ravel_copy[11] = 1.0       # -> saves 1.0 in unsharedData2 at [1, 1]
reshape_copy2[22] = 2.0    # -> saves 2.0 in unsharedData2 at [2, 2]
sharedData2 = mp.Array('d', ravel_copy)
sharedData2 = mp.Array('d', reshape_copy2)

answered May 3, 2021 at 16:39

dankal444

4,2081 gold badge28 silver badges41 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

MaPy · Accepted Answer · 2021-05-03 17:24:05Z

You can create a new multidimensional numpy array in each process that shares the same memory by using get_obj() method associated with Array, which returns the ctypes array that presents a buffer interface.

See the below example:

import ctypes as c
import numpy as np
import multiprocessing as mp


unsharedData2 = np.zeros((10, 10))
n, m = unsharedData2.shape[0], unsharedData2.shape[1]


def f1(mp_arr):
    #in each new process create a new numpy array as follows:
    arr = np.frombuffer(mp_arr.get_obj())
    b = arr.reshape((n, m))# mp_arr arr and b share the same memory
    b[2][1] = 3


def f2(mp_arr):
    #in each new process create a new numpy array as follows:
    arr = np.frombuffer(mp_arr.get_obj())
    b = arr.reshape((n, m)) # mp_arr arr and b share the same memory
    b[1][1] = 2


if __name__ == '__main__':
    mp_arr = mp.Array(c.c_double, n*m)
    p = mp.Process(target=f1, args=(mp_arr,))
    q = mp.Process(target=f2, args=(mp_arr,))
    p.start()
    q.start()
    p.join()
    q.join()
    arr = np.frombuffer(mp_arr.get_obj())
    b = arr.reshape((10, 10))
    print(b)
    '''
    [[0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
     [0. 2. 0. 0. 0. 0. 0. 0. 0. 0.]
     [0. 3. 0. 0. 0. 0. 0. 0. 0. 0.]
     [0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
     [0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
     [0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
     [0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
     [0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
     [0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
     [0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]]
    '''

Nathan · Accepted Answer · 2021-10-15 18:39:50Z

While answers have already been given for multiprocessing, an alternative exists using ray which is an alternative multiprocessing framework.

With ray you can place any object into read-only shared memory using obj_ref = ray.put(obj). The nice thing is that ray has built in support for zero-copy retrieval of numpy arrays from shared memory.

There is a little overhead using rays implementation of shared memory, but considering the arrays are so large this probably won't be a problem.

import numpy as np
import ray

@ray.remote
def function(arr, num: int):
    # array is automatically retrieved if a reference is passed to
    # a remote function, you could do this manually with ray.get(ref)
    return arr.mean() + num

if __name__ == '__main__':
    ray.init()
    # generate array and place into shared memory, return reference
    array_ref = ray.put(np.random.randn(55, 117, 256, 256))
    # multiple processes operating on shared array
    results = ray.get([function.remote(array_ref, i) for i in range(8)])
    print(results)

Johannes Kasimir · Accepted Answer · 2022-03-04 00:11:01Z

0

You can use Numba to parallel process the array. I don't know exactly what kind of processing you are planning to do. But it's likely possible to speed up with numba.


from numba import njit, prange

@njit(parallel=True)
def process(array):
    m, n, o, p = array.shape
    for i in prange(m):
        # process slices individually

    # or do something else
    return result

You might also want to look into numbas vecorize, guvectorize or stencil depending on your use case.

answered Mar 4, 2022 at 0:11

Johannes Kasimir

3662 silver badges3 bronze badges

Collectives™ on Stack Overflow

Multiprocessing - shared memory with multidimensional numpy array

4 Answers 4

Comments

Comments

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

Comments

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related