[Python] Provide pybind11 type casters

[pitrou]

Describe the enhancement requested

pybind11 allows defining custom type casters to translate between C++ classes and Python objects:
https://pybind11.readthedocs.io/en/stable/advanced/cast/custom.html

We already provide a C++ API to wrap and unwrap PyArrow objects from/to PyObject pointers. It would probably nicer to pybind11 users if we also provided a more automated facility.

A draft implementation of some type casters was done by @timkpaine here:
https://github.com/timkpaine/arrow-cpp-python-nocopy/blob/main/src/apn-python/caster.hpp

Component(s)

Python

[pitrou]

@AlenkaF @jorisvandenbossche

[timkpaine]

I copied this from @maartenbreddels https://github.com/vaexio/vaex-arrow-ext/blob/master/src/caster.hpp

It’s definitely useful to have this pybind utility, the only thing I think would be important is to make a big (bigger?) warning about ABI compatibility. Every 6 months or so I forget that if I’m writing my own C++ library that I want to bind out into python to transfer arrows back and forth, I can’t necessarily just naively use the pyarrow interfaces.

[pitrou]

Also cc @danepitkin

[alippai]

Shouldn't Apache Arrow use nanobind instead if possible? I guess the Arrow project is modern enough to use it (Python 3.8+, C++17 required)

[timkpaine]

s/shouldn't/couldn't

I think pybind is preferable given its popularity, but such an effort in pybind should be easy enough to translate into other binding libraries.

[pitrou]

Shouldn't Apache Arrow use nanobind instead if possible? I guess the Arrow project is modern enough to use it (Python 3.8+, C++17 required)

PyArrow uses neither. This issue is about better supporting users of pybind11, not using it in PyArrow.