Python -- Share a Numpy Array Between Processes?

Question

I'd like to use python's multiprocessing module to utilize a multi-core Linux server.

I need all processes to have read/write access to the same shared memory.

Instead of using a list or a queue, is it possible to have a multi-dimentional numpy array as the shared object?

Eric O. Lebigot · Accepted Answer · 2012-04-02 14:08:40Z

9

I think I know what you're looking for: https://bitbucket.org/cleemesser/numpy-sharedmem/issue/3/casting-complex-ndarray-to-float-in

There's a short description on the web page saying: A shared memory module for numpy by Sturla Molden and G. Varoquaux that makes it easy to share memory between processes in the form of NumPy arrays. Originally posted to SciPy-user mailing list.

I, myself am using it just that way. Sharing NumPy arrays between processes. Works very well for me.

edited Apr 2, 2012 at 14:08

Eric O. Lebigot

95.1k49 gold badges223 silver badges263 bronze badges

answered Apr 2, 2012 at 9:37

Robert

1312 silver badges3 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

mabalenk Over a year ago

Could you please update the above link? It doesn't lead to the code anymore. Thank you!

Samuel · Accepted Answer · 2018-06-07 13:29:23Z

6

Look at this. I doesn't seem easy, but it's doable.

Edit: Link rotted, I have linked to another copy.

edited Jun 7, 2018 at 13:29

answered Jul 5, 2011 at 6:14

Samuel

2,5002 gold badges19 silver badges21 bronze badges

1 Comment

oarfish Over a year ago

Link has rotted away.

J.J · Accepted Answer · 2015-05-10 05:50:42Z

1

I found that even if you do not modify your numpy array after fork()'ing a bunch of child processes, you will still see your RAM skyrocket as childprocesses copy-on-write the object for some reason.

You can limit (or totally alleviate?) this problem by setting

"yourArray.flags.writeable = False"

BEFORE fork()'ing/Pool()'ing which seems to keep the RAM used down, and is a LOT less hassle than the other methods :)

answered May 10, 2015 at 5:50

J.J

3,6172 gold badges33 silver badges37 bronze badges

1 Comment

raphaelauv Over a year ago

the correct syntax is : myarray.flags['WRITEABLE'] = False

Collectives™ on Stack Overflow

Python -- Share a Numpy Array Between Processes?

3 Answers 3

1 Comment

1 Comment

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

1 Comment

1 Comment

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related