pip-compile is taking forever for python3.13

[Agent-Hellboy]

Environment Versions

1. OS Type Ubuntu 20.04
2. Python version: Python 3.13.0a6+
3. pip version: pip 24.0 from /usr/local/lib/python3.13/site-packages/pip (python 3.13)
4. pip-tools version: pip-compile, version 7.4.1

Steps to replicate

➜ ~ time pip-compile -o requirements.txt requirements.in is taking forever
^C
Aborted!
pip-compile -o requirements.txt requirements.in 3093.59s user 98.14s system 569% cpu 9:20.79 total

content of requirements.in

PyQt5~=5.15.7
PyQt5-stubs~=5.15.6.0
grpc-stubs~=1.24.12
openpyxl-stubs~=0.1.25
pymongo-stubs~=0.2.0
types-PyYAML~=6.0.12.4
types-certifi~=2021.10.8.3
types-httplib2~=0.21.0.3
types-jsonschema~=4.17.0.3
types-paramiko~=2.12.0.3
types-protobuf~=4.21.0.4
types-python-dateutil~=2.8.19.6
types-requests~=2.28.11.8
types-retry~=0.9.9.1
types-setuptools~=65.7.0.4
types-tabulate~=0.9.0.0
types-urllib3~=1.26.25.4
types-six~=1.16.21.4
types-toml~=0.10.8.1
xmlschema~=2.1.1
protobuf~=5.26.1
tomli~=2.0.1
matplotlib~=3.8.4
Mako~=1.3.2
toml~=0.10.2

Expected result

should give result in few seconds
...

Actual result

No result
...

[webknjaz]

I'm guessing that the resolver is backtracking. Python 3.13 is not final yet so projects with C-extensions don't ship pre-compiled wheels for it. Pip is trying to compile whatever it finds and if it crashes (due to not yet being compatible with 3.13, which is what you're likely seeing), it tries to compile older versions. IIRC Qt things take a lot of time to compile.

First, verify that a simple pip install can handle it and if yes, then it's a problem with pip-tools. But it likely can't so you'll have to wait. I maintain a few projects with C-extensions (like aiohttp) and we also can't start supporting and publish pre-compiled wheels for pre-releases of CPython because the ABI is changing and the deps aren't yet caught up, and we can't test it in CI because of this even.

You'll likely have to wait until Python 3.13 final release in the fall before PyQt can run on that version.

[Agent-Hellboy]

Thanks for the explanation, actually matplotlib was the culprit.

[webknjaz]

Good to know! It was just a hunch but I'm glad you managed to pinpoint the root cause. I'm not sure what pip-tools could do differently to improve the situation. Perhaps, detecting that the Python runtime is not a stable release and issuing a warning on stderr with an explanation would be useful.. WDYT?

[Agent-Hellboy]

Hi @webknjaz , Thanks for mainting the tool with this much of passion, it's working but taking time
python3 -m piptools compile requirements.in -v 1241.40s user 39.43s system 514% cpu 4:08.89 total
I canceled at 9 min beacuse I thought it should take less time , it took 4min as most of library are cached.

btw, I have few open questions

numpy doesn't support 3.13 , how come this is working?

  Collecting numpy>=1.21 (from matplotlib~=3.8.4->-r requirements.in (line 23))
    Using cached numpy-1.26.4.tar.gz (15.8 MB)
    Installing build dependencies ... done
    Getting requirements to build wheel ... done
    Installing backend dependencies ... done
    Preparing metadata (pyproject.toml) ... done

installing from source?
i guess it was installing few matplotlib dependent from source that's why it was taking that much of time.

btw, where is it cached?

➜  pip ls /home/proshan/.cache/pip | grep numpy | grep ".tar.gz"

➜  pip cd ~
➜  ~ ls /home/proshan/.cache/pip | grep numpy | grep ".tar.gz"

➜  ~ sudo find / -name 'numpy*.tar.gz' 2>/dev/null

it will be helpful for me , if you explain how exactly this tool work or maybe point me to some stackoverflow answer or article

it has picked prebuilt wheel for below , how they are compitable

 Collecting PyQt5~=5.15.7 (from -r requirements.in (line 1))
    Using cached PyQt5-5.15.10-cp37-abi3-manylinux_2_17_x86_64.whl.metadata (2.1 kB)
  Collecting PyQt5-stubs~=5.15.6.0 (from -r requirements.in (line 2))
    Using cached PyQt5_stubs-5.15.6.0-py3-none-any.whl.metadata (3.8 kB)
  Collecting grpc-stubs~=1.24.12 (from -r requirements.in (line 3))
    Using cached grpc_stubs-1.24.12.1-py3-none-any.whl.metadata (4.1 kB)
  Collecting openpyxl-stubs~=0.1.25 (from -r requirements.in (line 4))
    Using cached openpyxl_stubs-0.1.25-py3-none-any.whl.metadata (1.7 kB)
  Collecting pymongo-stubs~=0.2.0 (from -r requirements.in (line 5))
    Using cached pymongo_stubs-0.2.0-py3-none-any.whl.metadata (3.3 kB)
  Collecting types-PyYAML~=6.0.12.4 (from -r requirements.in (line 6))
    Using cached types_PyYAML-6.0.12.20240311-py3-none-any.whl.metadata (1.5 kB)
  Collecting types-certifi~=2021.10.8.3 (from -r requirements.in (line 7))
    Using cached types_certifi-2021.10.8.3-py3-none-any.whl.metadata (1.4 kB)
  Collecting types-httplib2~=0.21.0.3 (from -r requirements.in (line 8))
    Using cached types_httplib2-0.21.0.5-py3-none-any.whl.metadata (1.3 kB)
  Collecting types-jsonschema~=4.17.0.3 (from -r requirements.in (line 9))
    Using cached types_jsonschema-4.17.0.10-py3-none-any.whl.metadata (1.6 kB)
  Collecting types-paramiko~=2.12.0.3 (from -r requirements.in (line 10))
    Using cached types_paramiko-2.12.0.3-py3-none-any.whl.metadata (1.3 kB)
  Collecting types-protobuf~=4.21.0.4 (from -r requirements.in (line 11))
    Using cached types_protobuf-4.21.0.7-py3-none-any.whl.metadata (1.3 kB)
  Collecting types-python-dateutil~=2.8.19.6 (from -r requirements.in (line 12))
    Using cached types_python_dateutil-2.8.19.20240311-py3-none-any.whl.metadata (1.8 kB)
  Collecting types-requests~=2.28.11.8 (from -r requirements.in (line 13))
    Using cached types_requests-2.28.11.17-py3-none-any.whl.metadata (1.3 kB)
  Collecting types-retry~=0.9.9.1 (from -r requirements.in (line 14))
    Using cached types_retry-0.9.9.4-py3-none-any.whl.metadata (1.4 kB)
  Collecting types-setuptools~=65.7.0.4 (from -r requirements.in (line 15))
    Using cached types_setuptools-65.7.0.4-py3-none-any.whl.metadata (1.3 kB)
  Collecting types-tabulate~=0.9.0.0 (from -r requirements.in (line 16))
    Using cached types_tabulate-0.9.0.20240106-py3-none-any.whl.metadata (1.6 kB)
  Collecting types-urllib3~=1.26.25.4 (from -r requirements.in (line 17))
    Using cached types_urllib3-1.26.25.14-py3-none-any.whl.metadata (1.7 kB)
  Collecting types-six~=1.16.21.4 (from -r requirements.in (line 18))
    Using cached types_six-1.16.21.20240425-py3-none-any.whl.metadata (1.5 kB)
  Collecting types-toml~=0.10.8.1 (from -r requirements.in (line 19))
    Using cached types_toml-0.10.8.20240310-py3-none-any.whl.metadata (1.5 kB)
  Collecting xmlschema~=2.1.1 (from -r requirements.in (line 20))
    Using cached xmlschema-2.1.1-py3-none-any.whl.metadata (8.1 kB)
  Collecting protobuf~=5.26.1 (from -r requirements.in (line 21))
    Using cached protobuf-5.26.1-cp37-abi3-manylinux2014_x86_64.whl.metadata (592 bytes)
  Collecting tomli~=2.0.1 (from -r requirements.in (line 22))
    Using cached tomli-2.0.1-py3-none-any.whl.metadata (8.9 kB)