Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

os.add_dll_directory required on CPython 3.8 and later #1532

Closed
zooba opened this issue May 21, 2021 · 2 comments
Closed

os.add_dll_directory required on CPython 3.8 and later #1532

zooba opened this issue May 21, 2021 · 2 comments

Comments

@zooba
Copy link
Contributor

zooba commented May 21, 2021

I haven't done the historical work, but at some point this code below was added/changed.

It's not mysterious - conda-forge has a patch to disable add_dll_directory and reenable finding DLLs on PATH. Because regular CPython does not have this patch, this change won't work in a range of scenarios where PATH is ignored.

So this needs the add_dll_directory call as well (or ideally, conda-forge would improve their patch, but I think that's unlikely right now).

pyzmq/zmq/__init__.py

Lines 65 to 96 in b33873c

def _libs_on_path():
"""context manager for libs directory on $PATH
Works around mysterious issue where os.add_dll_directory
does not resolve imports (conda-forge Python >= 3.8)
"""
if not sys.platform.startswith("win"):
yield
return
libs_dir = os.path.abspath(
os.path.join(
os.path.dirname(__file__),
os.pardir,
"pyzmq.libs",
)
)
if not os.path.exists(libs_dir):
# no bundled libs
yield
return
path_before = os.environ.get("PATH")
try:
os.environ["PATH"] = os.pathsep.join([path_before or "", libs_dir])
yield
finally:
if path_before is None:
os.environ.pop("PATH")
else:
os.environ["PATH"] = path_before

@zooba
Copy link
Contributor Author

zooba commented May 21, 2021

I see there's code higher up that ought to be doing the same thing. That makes this less urgent, but it does make this file pretty messy.

My other issues when I noticed this are covered by #1460

@zooba zooba closed this as completed May 21, 2021
@minrk
Copy link
Member

minrk commented May 25, 2021

There are two ways libzmq can be 'bundled' with pyzmq:

  1. externally built libzmq vendored in pyzmq.libs by delvewheel/auditwheel/delocate, in which case it is loaded as a regular DLL dependency of the backend at import time, and
  2. libzmq built by pyzmq itself as a Python Extension, in which case it's loaded via import zmq._libzmq before importing the backend

Option 2 was how we did it on Windows for a long time time, and what happens when pyzmq is built from source with libzmq unavailable on the system (~never on Windows). 1. is generally preferable, and, thanks to delvewheel, is how we do it now in almost all cases (I couldn't get it to work on Windows + PyPy for some reason, so that's the only remaining case where a pyzmq wheel bundles libzmq as an Extension).

Whatever advice you have for finding DLLs in pyzmq.libs that works consistently across Python versions and implementations would be welcome!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants