Build static module with Pybind11 and link against the interpreter

[allsey87]

I am embedding Python in an application and I would like to statically link against a Pybind11 module so that it is available without having to use dlopen etc.

I think I have to add a line in Python-*/Modules/Setup under the *static* section. I believe I will be able to hack something together, but I am wondering if there is a standard way of doing this?

In #466 @wjakob mentions that:

Python extension modules have to be dynamic.

But I think that is not entirely true, unless it is a limitation of Pybind11, right?

[allsey87]

I have tried adding the following line to my Setup.local

_MODULE_pybind11 path/to/some/libMODULE.a

I set the identifier of the module like this since Python links against the symbol PyInit_MODULE and Pybind11 produces the symbol PyInit__MODULE_pybind11, so by setting the identifier to _MODULE_pybind11 the two line up.

I confirmed that the symbol requested by Python exists in the libMODULE.a (statically linked Pybind11 module) but it seems to not be getting included. I am currently trying to bring in the whole archive by manipulating the linker flags as specified in tweaking compiler linker flags. I am currently trying with the following line but it still isn't quite working:

_MODULE_pybind11 -Xlinker --whole-archive path/to/some/libMODULE.a

Using nm, I can see libpython3.11a contains:

U PyInit__MODULE_pybind11

while libMODULE.a contains:

T PyInit__MODULE_pybind11

So it seems like it is just a question of getting that symbol from A to B...

[allsey87]

I have something somewhat working (somewhat being the keyword there). It is sufficient to just have a single line in Setup.local under *static* that contains the identifier of the package:

_MODULE_pybind11

This is sufficient to cause the interpreter library to be built with the unresolved symbol U PyInit__MODULE_pybind11, this symbol can then be provided during linking the executable that embeds the interpreter. Now, the catch is that the module has to be imported by the name _MODULE_pybind11 and not MODULE and I think this may be causing other issues too since although I can import the module, I cannot see anything on it (e.g., no __version__ field, etc).

[allsey87]

Ok, I overlooked that the module I was trying to build actually contains a mixture of Python code and C++ code. Hence the issue was that I had MODULE (Python code) and _MODULE_pybind11 (C++ code). The code of the C++ module itself defined the name _MODULE_pybind11 and it is used internally by MODULE.

I am still trying to patch everything up and get stuff working, but in summary, it seems that it is pretty straight forward to statically link a Pybind11 library to the Python interpreter.