Access numpy array from separate C process using shared memory

Question

I have a 1-D numpy array in memory

>>> x = np.arange(5)

I want to share this data with a separate and independent (not forked) C process on the same computer using shared memory.

I expect to do something like the following:

Allocate a new block of shared memory from within Python
Copy current data into that block
Get dtype, length, and global address of the array and pass these to a C process (for now, we'll pass to the C process over a command line interface)
Create an appropriately typed pointer within C to the memory address
Do some trivial computation in C

What is the best way to acheive these steps? There appear to be several Python solutions to allocate array data in shared memory. All examples I can find involve sharing between two Python processes rather than between Python and another language.

Minimal examples are greatly welcomed.

Too broad. But why not pass between Python processes and have one call the C functions? — too honest for this site
– too honest for this site, Commented Jan 19, 2016 at 16:31
This supports developers that strongly prefer to build standalone C applications without Python sandboxing. — MRocklin
– MRocklin, Commented Jan 19, 2016 at 16:35
Just provide them the framework. You need at least some Python headers anyway to get the layout of the stored object. And some helpers for synchronisation (e.g. mutex). — too honest for this site
– too honest for this site, Commented Jan 19, 2016 at 16:37
I plan to pass dtype, strides, and memory address as data. My hope is that this fully specifies the array so that they don't need Python headers. — MRocklin
– MRocklin, Commented Jan 19, 2016 at 17:47
Sorry, but that sounds more complicated than providing a basic framework (you have to pass that anyway). But not my project, so I'm out here. (I tend to ignore the fact quite some programmers are too lazy to learn something new. That because I don't want to make their problems mine.) — too honest for this site
– too honest for this site, Commented Jan 19, 2016 at 17:59

MRocklin · Accepted Answer · 2016-01-20 23:13:19Z

Here is a minimal example:

Python

import os

import posix_ipc
import numpy as np

x = np.arange(1000, dtype='i4')
f = posix_ipc.SharedMemory('test', flags=posix_ipc.O_CREAT, size=x.nbytes, read_only=False)

ff = os.fdopen(f.fd, mode='wb')
ff.write(x.data)
ff.close()  # flush doesn't work, but this does.

C

// shm.c
#include <stdlib.h>
#include <stdio.h>
#include <fcntl.h>
#include <sys/mman.h>

int main(int argc, char *argv[])
{
    int i, fd;
    int *data;

    fd = shm_open("test", O_RDONLY, 0);
    if (fd == -1)
        printf("Error!, bad file desciptor");

    data = mmap(NULL, sizeof(int), PROT_READ, MAP_PRIVATE, fd, 0);
    if (data == MAP_FAILED)
        printf("Error!, map failed!");

    for(i = 0; i < 1000; i++){
        printf("%d, ", (int)data[i]);
    }
    return 0;
}

$ gcc shm.c -lrt -o shm
$ ./shm 
0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, ...

Collectives™ on Stack Overflow

Access numpy array from separate C process using shared memory

1 Answer 1

Python

C

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Python

C

Comments

Your Answer

Sign up or log in

Post as a guest

Related