MWE for path error when using libpython310 on macOS

[speth]

This is a MWE for an issue building a package that uses an embedded Python interpreter by linking to libpython3.10.dylib and starting the interpreter from the Python C API. The problem is that in the embedded interpreter, sys.path refers to paths in the base conda environment, rather than the active conda environment corresponding to the libpython3.10.dylib that is linked. This then makes it impossible to import packages that are installed (only) in the active conda environment.

Examining sys.path

Relevant files:

• meta.yaml
• mwe.cpp
• test_mwe.sh

This example builds the following C++ program:

#include "Python.h"

int main(int argc, char** argv)
{
    Py_Initialize();
    PyRun_SimpleString(
        "import sys\n"
        "print('Hello, world')\n"
        "print('embedded sys.path:', sys.path)\n"
        "sys.stdout.flush()\n");
    return 0;
}

with the compiler command:

clang++ -I${CONDA_PREFIX}/include/python3.10 -L${CONDA_PREFIX}/lib mwe.cpp -o mwe -lpython3.10 -Wl,-rpath,${CONDA_PREFIX}/lib

This shows that the embedded interpreter's sys.path is (incorrectly) defined as:

['/Users/runner/miniforge3/lib/python310.zip',
 '/Users/runner/miniforge3/lib/python3.10',
 '/Users/runner/miniforge3/lib/python3.10/lib-dynload',
 '/Users/runner/miniforge3/lib/python3.10/site-packages']

While the python binary in this environment reports sys.path as:

['',
 '$PREFIX/lib/python310.zip',
 '$PREFIX/lib/python3.10',
 '$PREFIX/lib/python3.10/lib-dynload',
 '$PREFIX/lib/python3.10/site-packages']

(where the instances of $PREFIX are output substitutions introduced by Azure Pipelines).

Specific conditions required to trigger behavior

This error occurs when:

• Using a conda-forge feedstock
• With macOS x64 (Intel)
• With Python 3.10

Other versions of Python (3.8, 3.9, and 3.11) all work fine, as do all builds for Linux and Windows. I have also successfully built a minimal embedded Python test using the conda-forge build of Python 3.10 for macOS x64 outside of the conda-forge feedstock infrastructure (see repository and CI results here).

[conda-forge-webservices]

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

[bryanwweber]

@speth was this resolved upstream?

[speth]

Not to my knowledge. The issue I created (conda-forge/conda-forge.github.io#1996) is still open and got no response. I'm not sure how to update/re-run this to test it again.

[bryanwweber]

I asked around if there was someone who could help out with that issue. Let's see if it gets a response