Shared Non-Contiguous-Access Numpy Array

Question

I have a numpy array that I would like to share between a bunch of python processes in a way that doesn't involve copies. I create a shared numpy array from an existing numpy array using the sharedmem package.

import sharedmem as shm
def convert_to_shared_array(A):
    shared_array = shm.shared_empty(A.shape, A.dtype, order="C")
    shared_array[...] = A
    return shared_array

My problem is that each subprocess needs to access rows that are randomly distributed in the array. Currently I create a shared numpy array using the sharedmem package and pass it to each subprocess. Each process also has a list, idx, of rows that it needs to access. The problem is in the subprocess when I do:

#idx = list of randomly distributed integers

local_array = shared_array[idx,:]

# Do stuff with local array

It creates a copy of the array instead of just another view. The array is quite large and manipulating it first before shareing it so that each process accesses a contiguous range of rows like

local_array = shared_array[start:stop,:]

takes too long.

Question: What are good solutions for sharing random access to a numpy array between python processes that don't involve copying the array?

The subprocesses need readonly access (so no need for locking on access).

For whatever it's worth, this has nothing to do with it being shared memory. Numpy arrays have to be contiguous in memory, so any fancy indexing (e.g. indexing with an array of indicies) always returns a copy. This is true for any numpy array. Either work with smaller chunks of indicies or iterate, rather than slice. — Joe Kington
– Joe Kington, Commented Dec 19, 2011 at 21:17
@JoeKington - You're right. The problem is with numpy arrays in general. — Anthony Bak
– Anthony Bak, Commented Dec 22, 2011 at 19:24

David Cournapeau · Accepted Answer · 2012-01-03 06:12:23Z

1

Fancy indexing induces a copy, so you need to avoid fancy indexing if you want to avoid copies there is no way around it.

answered Jan 3, 2012 at 6:12

David Cournapeau

81.1k9 gold badges69 silver badges72 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Shared Non-Contiguous-Access Numpy Array

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related