Can't use holders around types with custom converters

[jagerman]

This fails for any T with a custom type converter (e.g. std::string, Eigen::MatrixXd, MyCustomConvertingType, etc.):

m.def("f", [](std::shared_ptr<T>) {})

The issue is that the holder type casters assume that type_caster<T> is a type_caster_base<T>, and make use of type_caster_base<T> constructors and internals. #786 makes it into a compile-time failure (before that it will result in strange run-time failures as we end up invoking a generic caster on an unregistered type).

[jagerman]

I don't see a nice way to implement this for all casters.

One possible idea for thought: type casters could opt-in to supporting casting to/from holders by providing static bool load_into(holder_type<T> &, handle src, bool convert) and static cast_from(const holder_type<T> &); then we can SFINAE on the existence of those methods when trying to cast a holder around a non-generic type.

[uentity]

@jagerman, I rebuild my code against latest master branch with your commit and static asserts didn't triggered. I think that's because I don't actually specify class holder. I use custom casting directly for shared_ptr< bs_numpy_array >, because It makes sense in one place -- make and hold a reference to an array on C++ side during corresponding numpy array lifetime in Python.

Here you can see what I do: url

[jagerman]

(URL tag arguments flipped; I do that all the time).

[jagerman]

Ah, you have your own type caster for a shared_ptr specifically, which is going to bypass the built-in generic holder casters I added the static asserts to.

[jagerman]

I think it's because of this part:

	operator sp_array*() { return &value; }
	operator sp_array&() { return value; }
	operator Array*() { return value.get(); }
	operator Array&() { return *value; }

	template <typename _T> using cast_op_type = pybind11::detail::cast_op_type<_T>;

so pybind could be getting a raw pointer to the shared_ptr, and trying to wrap that in a unique pointer. I think you can change this to just:

	operator sp_array() { return value; }
	template <typename> using cast_op_type = sp_array;

to just make a copy of the shared pointer (rather than a reference or pointer) when it needs to be accessed.

As for why you don't hit the static_assert, I think it's the nested shared pointers here, and I'll try to figure out a fix for that as well.

[uentity]

@jagerman thank for suggestion!
Did what you proposed, compiled fine, behaviour is the same as before -- #785 is not resolved.

How can I specify holder type with custom type caster? Is this what cast_op_type is for? And what is the purpose of this type casting operators inside type_caster?

[jagerman]

(Moving discussion back to #785).

A resolution to this bug (#787) would, I believe, make any registered holder work, so that when pybind tries to bind a holder type to a non-class type (e.g. a built-in string, Eigen array, or a user-provided custom converter) it would use the holder type to load the custom caster. (At least, that's what I have in mind in my suggestion above; we might go in some totally different direction, too).

[uentity]

BTW I'm in process of refactoring bs_nparray_traits implementation to make a composition instead of inheritance. Looking again at the code and come to conclusion that I don't need to build capsule to hold reference to C++ object during numpy array lifetime at all -- because my C++ array itself is a wrapper of array_t, so I can directly pass it as base to casted array. Am I right?

[jagerman]

Yes. The capsule use in the eigen code you got that from was because we return a numpy array which is directly referencing an eigen object's internal data, but that data is only valid so long as the eigen object stays alive.

[uentity]

While moving on with Python bindings for my class hierarchy I stumbled over another difficulty that I think is related to this discussion topic.

All my types are derived from base class objbase and I export 'em using conventional approach via pybind11::class_ with holder type being std::shared_ptr<T> (T is one of types in hierarchy). objbase itself is also exported because I want some Python types to derive from it.

Now the question is: how can I provide a fallback type caster that will allow me to pass any Python type to a function, taking std::shared_ptr<objbase> argument, but after all other registered converters to types, derived from objbase have been tried?

What I want: I have a function f(std::shared_ptr<objbase> param). If I pass an instance of object derived from objbase then I get what I pass. Otherwise if I pass some other (arbitrary) Python type instance then I want binding code to instantiate a simple wrapper class py_anyobj : public objbase (that is capable to carry any Python type), initialize it with Python instance and pass that into f.

I can provide an implementation of py_anyobj as follows:

class py_anyobj : public objbase {
protected:
	py::object pyobj_;

public:
	// inherit ctors
	using objbase::objbase;

	py_anyobj(const py::object& o) : pyobj_(o) {}
	py_anyobj(py::handle h) : pyobj_{ py::reinterpret_steal<py::object>(h) } {}

	py::object& pyobj() { return pyobj_; }
	const py::object& pyobj() const { return pyobj_; }

	// cast to py::object
	operator py::object() { return pyobj_; }
	operator py::object*() { return &pyobj_; }
};

Then, I can make custom type_caster for std::shared_ptr<py_anyobj> (I cannot make it for py_anyobj directly and specify std::shared_ptr<py_anyobj> as custom type holder because of #787):

template<> struct type_caster<std::shared_ptr<py_anyobj>> {
private:
	using sp_anyobj = std::shared_ptr<py_anyobj>;
	sp_anyobj value;

public:
	static handle cast(const sp_anyobj& src, return_value_policy, handle) {
		if(!src) return none().release();
		return src->pyobj().release();
	}

	bool load(handle src, bool convert) {
		value = std::make_shared<py_anyobj>(src);
		return true;
	}

	static PYBIND11_DESCR name() {
		return type_descr(_("py_anyobj"));
	}

	operator sp_anyobj() { return value; }
	template <typename> using cast_op_type = sp_anyobj;
};

But then what? OK, now I can pass any Python type to function taking sp_anyobj. But I want it to trigger for std::shared_ptr<objbase>! Is there a way to solve this with pybind11?

A nice addition would be backward C++ -> Python auto-convertion, i.e. when I return sp_anyobj from f, I should get captured Python instance back (if I return some custom instance, derived from objbase, then I receive that type as expected).

I had a semi-solution to this with boost::python earlier, it worked as I described in Python -> C++ way (most important to me), but not in the opposite (I got py_anyobj instances in Python). Cannot reproduce this with pybind11.
Maybe I'm looking in completely wrong direction?

[jagerman]

You'll want to make py_anyobj a trampoline class of objbase rather than a registered type; then you can declare objbase implicitly convertible from a py::object via the trampoline class. Working example below (I'll leave learning all the details of how this works up to you—I'm pretty sure everything used below is documented).

#include <pybind11/pybind11.h>

namespace py = pybind11;

class B {
public:
    virtual std::string say() { return "hi from B"; }
};

class A : public B {
public:
    std::string say() override { return "hi from A"; }
};

class PyB : public B {
public:
    using B::B;
    PyB() = default;
    PyB(py::object o) : myobj{o} { py::print("(PyB constructed from py::object)"); }
    std::string say() override { return "hi from a pyobject: " + (std::string) py::str(myobj); }
private:
    py::object myobj;
};

PYBIND11_PLUGIN(t787) {
    py::module m("t787");

    py::class_<B, PyB, std::shared_ptr<B>>(m, "B")
        .def(py::init<>())
        .def(py::init_alias<py::object>())
        ;
    py::class_<A, B, std::shared_ptr<A>>(m, "A")
        .def(py::init<>())
        ;

    py::implicitly_convertible<py::object, B>();

    m.def("foo", [](std::shared_ptr<B> b) { return "foo says: " + b->say(); });

    return m.ptr();
}

Edit: oops, forgot the output:

>>> from t787 import A, B, foo
>>> print(foo(B()))
foo says: hi from B
>>> print(foo(A()))
foo says: hi from A
>>> print(foo([1, 2, 3]))
(PyB constructed from py::object)
foo says: hi from a pyobject: [1, 2, 3]

[uentity]

One more situation considering the topic.

1. There is base class objbase exported conventionally via class_ with holder type std::shared_ptr<objbase>.
2. Derived class mytype_simple is using custom type caster for std::shared_ptr<mytype_simple> (cannot use type caster for mytype_simple and have std::shared_ptr holder).
3. Function f takes std::shared_ptr<objbase> argument.
4. How to pass instance of mytype_simple to f from Python?

Code.

#include <pybind11/pybind11.h>
#include <iostream>

namespace py = pybind11;
using namespace pybind11::literals;

/*-----------------------------------------------------------------------------
 *  objbase
 *-----------------------------------------------------------------------------*/
class objbase : public std::enable_shared_from_this< objbase > {
	int prop_ = 42;
public:
	virtual std::string name() const {
		return "objbase";
	}

	int prop() const { return prop_; }
};
using sp_obj = std::shared_ptr<objbase>;
using sp_cobj = std::shared_ptr<const objbase>;

// trampoline
template<class T = objbase>
class py_objbase : public T {
public:
	using T::T;

	std::string name() const override {
		PYBIND11_OVERLOAD(std::string, T, name);
	}
};

/*-----------------------------------------------------------------------------
 *  mytype_simple
 *-----------------------------------------------------------------------------*/
class mytype_simple : public objbase {
public:
	long value;

	using objbase::objbase;
	mytype_simple(long value_) : value(value_) {}

	std::string name() const override {
		return "mytype_simple";
	}
};

// custom type cast
NAMESPACE_BEGIN(pybind11)
NAMESPACE_BEGIN(detail)

template<> struct type_caster<std::shared_ptr<mytype_simple>> {
	std::shared_ptr<mytype_simple> value;

	static PYBIND11_DESCR name() {
		return type_descr(_("mytype_simple"));
	}

	bool load(handle src, bool) {
		/* Extract PyObject from handle */
		PyObject *source = src.ptr();
		/* Try converting into a Python integer value */
		PyObject *tmp = PyNumber_Long(source);
		if (!tmp)
			return false;
		/* Now try to convert into a C++ int */
		value = std::make_shared<mytype_simple>(PyLong_AsLong(tmp));
		Py_DECREF(tmp);
		/* Ensure return code was OK (to avoid out-of-range errors etc) */
		return !PyErr_Occurred();
	}

	static handle cast(const std::shared_ptr<mytype_simple>& src, return_value_policy, handle) {
		return PyLong_FromLong(src->value);
	}

	operator std::shared_ptr<mytype_simple>() { return value; }
	template< typename > using cast_op_type = std::shared_ptr<mytype_simple>;
};

NAMESPACE_END(detail)
NAMESPACE_END(pybind11)

PYBIND11_PLUGIN(example) {
	py::module m("example");

	// bind objbase
	py::class_<objbase, py_objbase<>, std::shared_ptr<objbase>>(m, "objbase")
		.def(py::init_alias<>())
		.def("name", &objbase::name)
	;

	m.def("test_objbase_pass", [](std::shared_ptr<objbase> obj) {
		std::cout << "Am I mytype_simple instance? " << std::boolalpha
			<< bool(std::dynamic_pointer_cast<mytype_simple>(obj)) << std::endl;
	});
	m.def("test_mytype_pass", [](std::shared_ptr<mytype_simple> obj) {
		std::cout << "Am I objbase instance? " << std::boolalpha
			<< bool(std::dynamic_pointer_cast<objbase>(obj)) << std::endl;
	});
}

Python test:

In [1]: import example as e

In [2]: e.test_mytype_pass(42L)
Am I objbase instance? true

In [3]: e.test_objbase_pass(42L)
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-3-304ca3cb5cbc> in <module>()
----> 1 e.test_objbase_pass(42L)

TypeError: test_objbase_pass(): incompatible function arguments. The following argument types are supported:
    1. (arg0: example.objbase) -> None

Invoked with: 42L

obviously not working.

All I can do is to add implicit cast:

py::implicitly_convertible< std::shared_ptr<mytype_simple>, objbase >();

But with this cast e.test_objbase_pass(42L) would segfault.

Is there any solution to this?

[flippmoke]

@jagerman I was looking into fixing this as I am attempting to use a std::shared_ptr<std::vector<T>>. I saw your original suggestion earlier here, but I wasn't entirely sure what all this would entail. I am still attempting to figure out the entire casting system but would love to help get together a PR.

[flippmoke]

Created an attempted solution for this in #1985.

[adamnovak]

I think I'm running into this issue with std::vector. Does it have a "custom converter" defined somewhere inside pybind11? I don't think I defined any custom converters for it.

I'm using it in the context of the Binder binding generator, which really wants to write bindings for any std::vector instantiations that your code uses, instead of using pybind11's automatic vector <-> Python list conversion (the custom converter?) and which provides a holder for the vector to pybind11:

https://github.com/RosettaCommons/binder/blob/788ab422f9e919478944d79d5890441a964dd1db/source/stl_binders.hpp#L34-L43

This seems to result in static assertion failed: Holder classes are only supported for custom types complaints out of the box (see RosettaCommons/binder#100), but I have found that using PYBIND11_MAKE_OPAQUE() on the vector instantiations causes those issues to go away, and gets me apparently-working bindings.

Am I actually running into this issue? Is PYBIND11_MAKE_OPAQUE() a know working workaround, or is it likely to be secretly leaking memory or something?