pybind11 return numpy array of objects

Question

Using pybind11 C++ API and python3, how can we properly create a numpy array of objects (i.e. unicode strings) in the C++ implementation and return it back to python? What is the exact memory layout of the underlying data array passed into pybind11::array()? How exactly do we need to manage memory, i.e. delete/free?

Note that this is necessary because we want to use that array of strings, in conjunction with other POD arrays, in pandas DataFrame creation.

I can't help with pybind11. But numpy arrays can be created with a function the numpy C API. Array layouts are all basically the same, attributes plus a data buffer. For an object type the data buffer contains pointers to objects elsewhere in memory. WIth a unicode dtype that buffer actually contains the strings (padded to a specified length). You may have to study the numpy docs. — hpaulj
– hpaulj, Commented Jul 16, 2017 at 4:00
thanks - it seems like the best way is to manually create the buffer with objects and then figure out how pybind11 can manage memory associated with the underlying array as well as all the objects... — joe-ts
– joe-ts, Commented Jul 18, 2017 at 1:53

joe-ts · Accepted Answer · 2017-07-20 00:04:02Z

3

Turns out that it is necessary to:

create an array of PyObject pointers, fill the array, i.e.

auto* pbuf = new PyObject*[arraySize]; // or create via pybind11 API...
pbuf[0] = <new object...>
pbuf[1] = <new object...>
etc.

create an "object" py::array() with a capsule:

py::capsule freeWhenDone(pbuf, [](void* pp) {
        delete [] (PyObject*)pp; // or else properly free the pbuf memory
    });

arr = py::array(py::dtype("object"),
    shape, strides, pbuf, freeWhenDone);

answered Jul 20, 2017 at 0:04

joe-ts

5531 gold badge5 silver badges8 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

olq_plo Over a year ago

Do you also know a solution which does not need the freeWhenDone? I suppose if the memory is malloc'd by pybind11, py::array can use the normal free'ing.

joe-ts Over a year ago

The way I was able to get it to work without leaking is with a capsule. Moreover, when inner objects are python strings, I had to free each individually allocated object. There may be other solutions with the newer version of pybind11.

Collectives™ on Stack Overflow

pybind11 return numpy array of objects

1 Answer 1

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related