Embedding a python/cython numpy array: Passing numpy array to c pointer

Question

I am trying to pass a cython-numpy array to a c struct. Which works sometimes but crashes if the array becomes too 'large'. While this is not really large (like <100k double type entries) and I cannot figure out where this limit is coming from. A few code snippets (full code attached).

Building on Win7, Cython 0.28.4, Python 3.6, mingw64

I tried this with different lengths of the array. The latest thing I discovered that it always crashes if the array length is larger than 2**16-512 entries. But I have no clue why.

The cython file:

//cy_file.pyx
//...
cdef public struct Container:
    double* np_array
// ...
cdef public void create_np_array(Container *container):
  cdef numpy.ndarray[numpy.float_t, ndim=1, mode = 'c'] np_array
  # just create a numpy array 
  longarray = np.linspace(1.0, 5.0, 100000)
  np_array = numpy.ascontiguousarray(numpy.array(longarray), dtype=float)
  container.np_array = <double*> np_array.data

And for the c-file:

//c_file.c
#include "Python.h"
#include "cy_file.h"

struct Container container;
//...
Py_Initialize();
PyInit_cy_file();
// and call the cython function that fills up the numpy array
create_np_array(&container, start, stop, n_elements, n);
// shutdown of python interpreter
Py_Finalize();

// *** here comes the crash if the array longarray is 'too long' ***
container.np_array[0]

Can anyone give me a hint what goes wrong here or how to debug?

Thanks and cheers, Tim

DavidW · Accepted Answer · 2019-01-14 17:38:29Z

0

The memory is owned by the Numpy array and gets freed when the reference count of the numpy array drops to zero, which is most likely at the end of create_np_array. Failing that Py_Finalize() attempts to free all remaining Python objects.

Your attempt to access this memory is always invalid - the fact that it only fails for arrays of certain sizes is just "luck".

There isn't any one good solution, but here are some suggestions:

Use Py_INCREF to manually increase the reference count of the owning numpy array (and then decrease it again manually when you are done with the memory it holds) so that it is not destroyed at the end of the function. Make sure you keep your memory accesses before Py_Finalize.
Handle the memory allocation in your C code and assign it to the Numpy array so that the numpy array does not own it using PyArray_SimpleNewFromData. You must be careful that the Numpy array does not outlive the memory.

answered Jan 14, 2019 at 17:38

DavidW

31.2k7 gold badges64 silver badges99 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

ToolTim Over a year ago

Thanks for pointing towards the right direction for me. I found some additional discussions on stackoverflow regarding PyArray_SimpleNewFromData, this one was particularly helpful: link

ToolTim Over a year ago

Another comment with helpful insight to the topic can be found here: link

ToolTim · Accepted Answer · 2019-03-06 16:53:23Z

0

Ok, finally found the answer with the helpful hints:

The idea is best described in a blog post including a complete code sample on github. So I just refer to this post here which answers the question in great detail.

Thanks!

answered Mar 6, 2019 at 16:53

ToolTim

295 bronze badges

1 Comment

Wai Ha Lee Over a year ago

This is a link-only answer at the moment. Could you edit it in such a way that if the link dies, your answer is still useful for others?

Collectives™ on Stack Overflow

Embedding a python/cython numpy array: Passing numpy array to c pointer

2 Answers 2

2 Comments

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related