ndarray data ownership question

[NavreetGill]

Hi,

Lets say I have a

boost::shared_ptrstd::vector vp;

What is the best way to construct a numpy array and return it?

I am currently using:

std::vector &vec(*vp);

np::from_data(vec.data(), np::dtype::get_builtin() , p::make_tuple(vec.size()), p::make_tuple(sizeof(int)), owner);

The question is: what should I put for owner? I'd like to be able to call vp.reset() in my C++ code [and therefore giving up ownership] and have python manage the memory as long as it needs to (and clean up if the python object is ever del'ed).

I am assuming having a p::object() on the stack, but I am guessing that it is probably not a good thing (as it will destruct when my function ends). So, I can create a "dummy" owner and have it linger around in my c++ code, but that'll keep objects around longer than they are needed (essentially leaking memory, w/o technically doing so).

Thanks in advance.

[TallJimbo]

What you want to do here is create an opaque Python object that does nothing but hold your shared_ptr, and destroy it when the Python object goes out of scope. Happily, you can do that with a Python CObject (or Capsule object, I believe, in Python 3), so you don't have to go to all of the work of creating a Boost.Python class just to create an owner. The ndarray project, which is a higher-level Python/C++ array conversion library that uses Boost.NumPy to access the NumPy C-API, does something very much like this here:

https://github.com/ndarray/ndarray/blob/master/include/ndarray/bp/Array.h#L33

It uses an ndarray::Manager::Ptr (a typedef to boost::intrusive_ptr) instead of a shared_ptr, but the idea is the same. The most important part is the call to PyCObject_FromVoidPtr; the rest is mostly just an attempt to avoid cycles of shared_ptrs that hold CObjects that hold shared_ptrs.

[NavreetGill]

OK, so essentially, the destroyManager function will get called when the objected is del'ed in python (or I guess when the GC decides to call it).

If I look closely, it seems like I will have to have different destroyManager functions depending on the type. I will probably use templates here.

I wish I had looked at your ndarray/Vector class earlier. It would've helped out.

I still think that the use case I mentioned might be good to add in the library (converting a shared_array<T> or shared_ptr<vector<T> > to a ndarray and transferring ownership).

[TallJimbo]

Yeah, I agree that it'd be worthwhile to add this functionality to Boost.NumPy. I'll keep this issue open to remind me to do that.

[7starsea]

My typical use case is:
namespace np = boost::python::numpy;
namespace bp = boost::python;
np::dtype dtype = np::dtype::get_builtin();
///creating np_shape, np_stride
data = new double[200];
arr = np::from_data( data_, dtype, np_shape, np_stride, bp::object() );_

and then the variable arr is returned to python and will be deleted in python, and I am experiencing a memory leak for the data_. How could I avoid the memory leak? @TallJimbo Thanks so much.

[7starsea]

Never mind, I solved the problem. For anyone who are experiencing the same memory-leaking, here is my solution.
We need to add a deconstructor:
boost::python::handle<> h(::PyCapsule_New((void *)data, NULL, (PyCapsule_Destructor)&destroyManagerCObject));
arr = np::from_data( data_, dtype, np_shape, np_stride, bp::object(h) );_
where
inline void destroyManagerCObject(PyObject* obj) {
double * b = reinterpret_cast<double*>( PyCapsule_GetPointer(obj, NULL) );
if(b){ delete [] b; }
}

Hope it helps.

[PhilippeCarphin]

Hope it helps.

It sure does help! This is an application of what 7starsea said to do boostorg/python#97 (comment)