virtual copying

[ripaul]

Hi, I am trying to write bindings for a bunch of classes, that implement a common interface A. Now this interface declares a function std::unique_ptr<A> copy() const which is supposed to create a copy of the current object. Following the documentation on trampoline classes etc., I came up with this code:

#include <pybind11/pybind11.h>

#include <memory>

struct A { 
    virtual ~A() = default;

    virtual std::unique_ptr<A> copy() const = 0;
};

template<typename Base>
struct Trampoline : public Base { 
    std::unique_ptr<A> copy() const { 
        PYBIND11_OVERRIDE_PURE(
            std::unique_ptr<A>,
            Base,
            copy
        );
    }
};

struct B : public A {
    std::unique_ptr<A> copy() const {
        return std::make_unique<B>(*this);
    }
};

void foo(A* a) {
    auto b = a->copy();
}

namespace py = pybind11;

PYBIND11_MODULE(main, m) {
    py::class_<A, Trampoline<A>>(m, "A")
        .def(py::init<>())
        .def("copy", &A::copy);

    py::class_<B, Trampoline<B>, A>(m, "B")
        .def(py::init<>())
        .def("copy", &B::copy);

    m.def("foo", &foo);
}

however, this doesn't compile. Instead I get the following error message, which I am having trouble to understand:

In file included from /test/main.cpp:1:
In file included from /test/pybind11/include/pybind11/pybind11.h:13:
In file included from /test/pybind11/include/pybind11/detail/../attr.h:13:
/test/pybind11/include/pybind11/cast.h:907:12: error: no matching function for call to 'cast_op'
    return cast_op<T>(load_type<T>(handle));
           ^~~~~~~~~~
/test/pybind11/include/pybind11/cast.h:961:12: note: in instantiation of function template specialization 'pybind11::cast<std::unique_ptr<A, std::default_delete<A> >, 0>' requested here
    return cast<T>(object);
           ^
/test/pybind11/include/pybind11/cast.h:991:22: note: in instantiation of function template specialization 'pybind11::cast<std::unique_ptr<A, std::default_delete<A> > >' requested here
    return pybind11::cast<T>(std::move(o)); }
                     ^
/test/main.cpp:14:9: note: in instantiation of function template specialization 'pybind11::detail::cast_safe<std::unique_ptr<A, std::default_delete<A> > >' requested here
        PYBIND11_OVERRIDE_PURE(
        ^
/test/pybind11/include/pybind11/pybind11.h:2524:5: note: expanded from macro 'PYBIND11_OVERRIDE_PURE'
    PYBIND11_OVERRIDE_PURE_NAME(PYBIND11_TYPE(ret_type), PYBIND11_TYPE(cname), #fn, fn, __VA_ARGS__)
    ^
/test/pybind11/include/pybind11/pybind11.h:2488:9: note: expanded from macro 'PYBIND11_OVERRIDE_PURE_NAME'
        PYBIND11_OVERRIDE_IMPL(PYBIND11_TYPE(ret_type), PYBIND11_TYPE(cname), name, __VA_ARGS__); \
        ^
/test/pybind11/include/pybind11/pybind11.h:2455:38: note: expanded from macro 'PYBIND11_OVERRIDE_IMPL'
            return pybind11::detail::cast_safe<ret_type>(std::move(o));                           \
                                     ^
/test/pybind11/include/pybind11/detail/init.h:61:68: note: in instantiation of member function 'Trampoline<A>::copy' requested here
inline Class *construct_or_initialize(Args &&...args) { return new Class(std::forward<Args>(args)...); }
                                                                   ^
/test/pybind11/include/pybind11/detail/init.h:199:31: note: in instantiation of function template specialization 'pybind11::detail::initimpl::construct_or_initialize<Trampoline<A>, 0>' requested here
            v_h.value_ptr() = construct_or_initialize<Alias<Class>>(std::forward<Args>(args)...);
                              ^
/test/pybind11/include/pybind11/pybind11.h:1429:14: note: in instantiation of function template specialization 'pybind11::detail::initimpl::constructor<>::execute<pybind11::class_<A, Trampoline<A> >, 0>' requested here
        init.execute(*this, extra...);
             ^
/test/main.cpp:36:10: note: in instantiation of function template specialization 'pybind11::class_<A, Trampoline<A> >::def<>' requested here
        .def(py::init<>())
         ^
/test/pybind11/include/pybind11/cast.h:54:73: note: candidate template ignored: substitution failure [with T = std::unique_ptr<A, std::default_delete<A> >]: no member named 'cast_op_type' in 'pybind11::detail::type_caster<std::unique_ptr<A, std::default_delete<A> >, void>'
template <typename T> typename make_caster<T>::template cast_op_type<T> cast_op(make_caster<T> &caster) {
                      ~~~~~~~~                                          ^
/test/pybind11/include/pybind11/cast.h:58:1: note: candidate template ignored: substitution failure [with T = std::unique_ptr<A, std::default_delete<A> >]: no member named 'cast_op_type' in 'pybind11::detail::type_caster<std::unique_ptr<A, std::default_delete<A> >, void>'
cast_op(make_caster<T> &&caster) {
^
...

The bit I am understanding is, that pybind seems to be having some trouble casting some handle to std::unique_ptr<A>, doesn't it? What do I need to do to make this work? Any help is much appreciated :)

[EricCousineau-TRI]

Ah, nuts 🤦 I believe I have a fix for this on our fork, been meaning to PR it:
https://github.com/RobotLocomotion/pybind11/blob/5a0552c73265b5ad3fddb6581f7dd1cb91497f57/include/pybind11/cast.h#L799-L803

Lemme see if I can PR this in next few min.

[EricCousineau-TRI]

Ah derp. While yes, the cast_op_type is part of it, the issue is actually a bit more deep-rooted:
You must be able to cast from a Python object to unique_ptr<> to support PYBIND11_OVERLOAD, which is not supported in base pybind11.

In this case, you may want to try out @rwgk's smart_holder branch, if you believe the functionality is what you need.
https://github.com/pybind/pybind11/blob/smart_holder/README_smart_holder.rst

(Our fork has it too, but it's way more messy.)

[EricCousineau-TRI]

Ah, and this is the branch where I repro'd your error:
master...EricCousineau-TRI:discussion-3487

[ripaul]

Hey Eric! Thank you very much for shedding some light on the issue for me! The linked branch seems promising, I will try this out.

Based on your clarifications, I realized that the issue indeed is resolved when I just use raw pointers. Now I'm wondering how "ownership" of the copied object would be handled? That is, is it safe to just return a raw pointer here and will Python's reference counting take care of deallocating the copy properly? I'm not very familiar with the internal workings of Python and pybind.

[rwgk]

Quick comment, without having looked much:

If you use the smart_holder branch with your unique_ptr interface, the lifetime handling will definitely be correct. The trampoline situation is heavily exercised on the smart_holder branch.

It may also work with a raw pointer, I'm not sure. If you create a PR (please tag me if you do) I could quickly run it with ASAN.