0

I would like to access an existing NumPy array from a subprocess using the multiprocessing module without copying the array to a shared memory object. Apparently, multiprocessing.Array creates such a shared memory array, but I can't seem to be able to point the array to an existing numpy.ndarray object. This is critical, because the existing array can be quite large (up to a couple of GB), so I definitely need to avoid any copy operations.

Here's what I've tried so far:

import multiprocessing as mp
import numpy as np


def f(x, idx):
    """Dummy function to manipulate an array."""
    x[idx] = 999


a = np.array([1.2, 15.8, 10.3, 7.4, -44.9])
b = mp.Array("d", a)  # apparently this creates a copy of a in b
print("Original array:".rjust(28, " "), a)

f(a, 0)
print("Change a[0] in main process:".rjust(28, " "), a)

p = mp.Pool(1)

p.apply_async(f, args=(b, 4))
print("Change b[4] in subprocess:".rjust(28, " "), np.frombuffer(b.get_obj()))

Ideally, I'd like a and b refer to the same underlying numbers, but apparently this is not working. Interestingly, b is also not changed by p.apply_async(f, args=(b, 4)) - this is probably not related to the original question, but I'd still like to understand why.

4
  • Let me know if this thing worked :D Commented May 8, 2018 at 14:07
  • I don't think this is a duplicate, since in this question unshared_arr and arr do not refer to the same address in memory, which is exactly what I'd like to do. Commented May 8, 2018 at 14:34
  • From the docs it looks like mp.Array creates a ctypes array in shared memory, and initializes it from values in a. It isn't storing or sharing a pointer to the a object. So b is not just a copy, it's not a numpy array; it's a different kind of data structure. Note in the docs it can intialized from a range, a list and Array.array. Commented May 8, 2018 at 16:34
  • Something went wrong with marking this as a duplicate - the wrong question is linked, could someone please correctly link to stackoverflow.com/questions/42037545/…? Commented May 9, 2018 at 10:05

0

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.