[Bug]: python packages incompatible with poetry

[mtxvp]

Problem description

Setup: I have a python project setup in Termux with poetry that depends on packages like cryptography, numpy, lxml and pandas.

These libraries are installed using "pkg install python-cryptography" type commands, to make it available globally.

Poetry is configured to include "system-site-packages", so it can take advantage of globally installed packages rather than trying to install and compile them itself.

Problem" When running "poetry update", poetry thinks that globally installed package is different from the one in PiPy, and tries to "Update" it, causing compilation failures in my limited environment (the main reason why python-cryptography package is provided in the first place).

Root-cause: Managed to narrow it down to logic in src/poetry/core/packages/specification.py:is_same_source_as(), that thinks package is different, because source_url is different: None in PiPy vs "/home/builder/.termux-build/python-cryptography/src" in the Termux package.

Suggested fix: When manually removing direct_url.json from the respective folder, poetry happily accepts already installed package and is able to complete the update command successfully. Is it possible to adjust build procedure for all python packages to exclude addition of direct_url.json file in them?

What steps will reproduce the bug?

• Install python-cryptography package
• Create an empty python project with cryptography dependency of the same version
• Create virtual environment for it and activate it
• Install poetry in the environment
• Run "poetry config virtualenvs.options.system-site-packages true" for vritualenv to allow seeing globally installed pip packages.
• Run "poetry update". Command will fail, trying to compile the cryptography wheel.

What is the expected behavior?

Successful completion of "poetry update"

System information

Termux Variables:
TERMUX_APK_RELEASE=F_DROID
TERMUX_APP_PACKAGE_MANAGER=apt
TERMUX_APP_PID=25611
TERMUX_IS_DEBUGGABLE_BUILD=0
TERMUX_MAIN_PACKAGE_FORMAT=debian
TERMUX_VERSION=0.118.1
TERMUX__USER_ID=0
Packages CPU architecture:
aarch64
Subscribed repositories:
# sources.list
deb https://mirrors.medzik.dev/termux/termux-main stable main
# tur-repo (sources.list.d/tur.list)
deb https://tur.kcubeterm.com tur-packages tur tur-on-device tur-continuous
# root-repo (sources.list.d/root.list)
deb https://mirrors.medzik.dev/termux/termux-root root stable
Updatable packages:
abseil-cpp/stable 20240116.2-1 aarch64 [upgradable from: 20240116.2]
bash/stable 5.2.37 aarch64 [upgradable from: 5.2.32]
binutils-bin/stable 2.43.1 aarch64 [upgradable from: 2.42]
binutils-libs/stable 2.43.1 aarch64 [upgradable from: 2.42]
blk-utils/stable 2.40.2-1 aarch64 [upgradable from: 2.39.3-4]
c-ares/stable 1.34.1 aarch64 [upgradable from: 1.33.0]
ca-certificates/stable 1:2024.09.24 all [upgradable from: 1:2024.07.02]
clang/stable 19.1.1-1 aarch64 [upgradable from: 18.1.8]
command-not-found/stable 2.4.0-46 aarch64 [upgradable from: 2.4.0-37]
coreutils/stable 9.5-3 aarch64 [upgradable from: 9.5]
curl/stable 8.10.1-1 aarch64 [upgradable from: 8.9.1]
dpkg/stable 1.22.6-1 aarch64 [upgradable from: 1.22.6]
findutils/stable 4.10.0 aarch64 [upgradable from: 4.9.0-2]
fribidi/stable 1.0.16 aarch64 [upgradable from: 1.0.15]
git/stable 2.47.0 aarch64 [upgradable from: 2.46.0]
glib/stable 2.80.5-2 aarch64 [upgradable from: 2.80.4]
harfbuzz/stable 10.0.1 aarch64 [upgradable from: 9.0.0]
ldns/stable 1.8.4 aarch64 [upgradable from: 1.8.3-2]
libblkid/stable 2.40.2-1 aarch64 [upgradable from: 2.39.3-4]
libc++/stable 27b aarch64 [upgradable from: 27]
libcairo/stable 1.18.2 aarch64 [upgradable from: 1.18.0]
libcompiler-rt/stable 19.1.1-1 aarch64 [upgradable from: 18.1.8]
libcurl/stable 8.10.1-1 aarch64 [upgradable from: 8.9.1]
libexpat/stable 2.6.3 aarch64 [upgradable from: 2.6.2]
libgcrypt/stable 1.11.0 aarch64 [upgradable from: 1.10.3-2]
libjpeg-turbo/stable 3.0.4 aarch64 [upgradable from: 3.0.3]
libllvm/stable 19.1.1-1 aarch64 [upgradable from: 18.1.8]
liblzma/stable 5.6.3 aarch64 [upgradable from: 5.6.2]
libmount/stable 2.40.2-1 aarch64 [upgradable from: 2.39.3-4]
libnghttp2/stable 1.63.0 aarch64 [upgradable from: 1.62.1]
libnghttp3/stable 1.6.0 aarch64 [upgradable from: 1.4.0]
libpng/stable 1.6.44 aarch64 [upgradable from: 1.6.43]
libraqm/stable 0.10.2 aarch64 [upgradable from: 0.10.1]
libsmartcols/stable 2.40.2-1 aarch64 [upgradable from: 2.39.3-4]
libtiff/stable 4.7.0 aarch64 [upgradable from: 4.6.0]
libunbound/stable 1.21.1-1 aarch64 [upgradable from: 1.21.0]
libuuid/stable 2.40.2-1 aarch64 [upgradable from: 2.39.3-4]
lld/stable 19.1.1-1 aarch64 [upgradable from: 18.1.8]
llvm/stable 19.1.1-1 aarch64 [upgradable from: 18.1.8]
mc/stable 4.8.32 aarch64 [upgradable from: 4.8.31-1]
nano/stable 8.2 aarch64 [upgradable from: 8.0]
ncurses-ui-libs/stable 6.5.20240831-1 aarch64 [upgradable from: 6.4.20231001-5]
ncurses/stable 6.5.20240831-1 aarch64 [upgradable from: 6.4.20231001-5]
ndk-sysroot/stable 27b aarch64 [upgradable from: 27-1]
nodejs-lts/stable 20.17.0 aarch64 [upgradable from: 20.15.1]
openssh-sftp-server/stable 9.9p1-4 aarch64 [upgradable from: 9.8p1-4]
openssh/stable 9.9p1-4 aarch64 [upgradable from: 9.8p1-4]
openssl/stable 1:3.3.2 aarch64 [upgradable from: 1:3.3.1]
pinentry/stable 1.3.1 aarch64 [upgradable from: 1.3.0-1]
ripgrep/stable 14.1.1 aarch64 [upgradable from: 14.1.0]
rust-std-aarch64-linux-android/stable 1.81.0-1 all [upgradable from: 1.80.0]
termux-auth/stable 1.5.0 aarch64 [upgradable from: 1.4-2]
termux-tools/stable 1.43.6 all [upgradable from: 1.43.3]
util-linux/stable 2.40.2-1 aarch64 [upgradable from: 2.39.3-4]
xz-utils/stable 5.6.3 aarch64 [upgradable from: 5.6.2]
zstd/stable 1.5.6-2 aarch64 [upgradable from: 1.5.6]
termux-tools version:
1.43.3
Android version:
13
Kernel build information:
Linux localhost 4.19.113-27095354 #1 SMP PREEMPT Thu May 16 18:40:15 KST 2024 aarch64 Android
Device manufacturer:
samsung
Device model:
SM-G981W
LD Variables:
LD_LIBRARY_PATH=
LD_PRELOAD=/data/data/com.termux/files/usr/lib/libtermux-exec.so

[licy183]

I'm not sure whether this is a bug or feature of poetry or pip. The same error will happen if you manually install prebuilt python wheels using pip.

Ubuntu 22.04:

wget https://files.pythonhosted.org/packages/ac/7e/ebda4dd4ae098a0990753efbb4b50954f1d03003846b943ea85070782da7/cryptography-43.0.1-cp39-abi3-manylinux_2_28_x86_64.whl
pip install cryptography-43.0.1-cp39-abi3-manylinux_2_28_x86_64.whl
cat ~/.local/lib/python3.10/site-packages/cryptography-43.0.1.dist-info/direct_url.json
# {"archive_info": {}, "url": "file:///home/builder/cryptography-43.0.1-cp39-abi3-manylinux_2_28_x86_64.whl"}

[fornwall]

Information about the direct_url.json file: https://packaging.python.org/en/latest/specifications/direct-url/

[fornwall]

I think we should get rid of direct_url.json (either by removing it, or specifying flags to pip so it's never created) - that fixes this issue and aligns with what at least debian/ubuntu and fedora does (for example, check dpkg -L python3-scipy | grep direct_url.json in ubuntu, compared to dpkg -L python-scipy | grep direct_url.json in termux.

Related discussion: https://discuss.python.org/t/pep-610-usage-guidelines-for-linux-distributions/4012