Can't build on Debian 12 (bookworm) due to missing libssl.so.1.1 (IDFGH-14722)

[dernasherbrezon]

Answers checklist.

• I have read the documentation ESP-IDF Programming Guide and the issue is not addressed there.
• I have updated my IDF branch (master or release) to the latest version and checked that the issue is present there.
• I have searched the issue tracker for a similar issue and not found a similar issue.

IDF version.

v5.4

Operating System used.

Linux

How did you build your project?

Command line with idf.py

If you are using Windows, please specify command line type.

None

What is the expected behavior?

Successful build.

What is the actual behavior?

Failure to build the project based on ESP-IDF.

Steps to reproduce.

1. Use Debian bookworm
2. Download any esp-idf project. I was using mine: https://github.com/dernasherbrezon/lora-at
3. Run idf.py build

Build or installation Logs.

[825/1162] Generating x509_crt_bundle
FAILED: esp-idf/mbedtls/x509_crt_bundle /home/pi/git/lora-at/build/esp-idf/mbedtls/x509_crt_bundle 
cd /home/pi/git/lora-at/build/esp-idf/mbedtls && /home/pi/.espressif/python_env/idf5.4_py3.11_env/bin/python /home/pi/esp/esp-idf/components/mbedtls/esp_crt_bundle/gen_crt_bundle.py --input /home/pi/esp/esp-idf/components/mbedtls/esp_crt_bundle/cacrt_all.pem /home/pi/esp/esp-idf/components/mbedtls/esp_crt_bundle/cacrt_local.pem -q --max-certs 200
The cryptography package is not installed.Please refer to the Get Started section of the ESP-IDF Programming Guide for setting up the required packages.
Traceback (most recent call last):
  File "/home/pi/esp/esp-idf/components/mbedtls/esp_crt_bundle/gen_crt_bundle.py", line 23, in <module>
    from cryptography import x509
  File "/home/pi/.espressif/python_env/idf5.4_py3.11_env/lib/python3.11/site-packages/cryptography/x509/__init__.py", line 7, in <module>
    from cryptography.x509 import certificate_transparency, verification
  File "/home/pi/.espressif/python_env/idf5.4_py3.11_env/lib/python3.11/site-packages/cryptography/x509/certificate_transparency.py", line 11, in <module>
    from cryptography.hazmat.bindings._rust import x509 as rust_x509
ImportError: libssl.so.1.1: cannot open shared object file: No such file or directory

More Information.

The newest version of Debian is using libssl 3. The installed python3-cryptography package inside ESP-IDF is version "43.0.3". I tried manually installing latest python3-cryptography using suggestion from #15207. But the build failed with the following:

The following Python requirements are not satisfied:
Requirement 'cryptography<44,>=2.1.4' was not met. Installed version: 44.0.1
To install the missing packages, please run "install.sh"
Diagnostic information:
    IDF_PYTHON_ENV_PATH: /home/pi/.espressif/python_env/idf5.4_py3.11_env
    Python interpreter used: /home/pi/.espressif/python_env/idf5.4_py3.11_env/bin/python
Constraint file: /home/pi/.espressif/espidf.constraints.v5.4.txt
Requirement files:
 - /home/pi/esp/esp-idf/tools/requirements/requirements.core.txt
Python being checked: /home/pi/.espressif/python_env/idf5.4_py3.11_env/bin/python

ESP-IDF v5.4

[fhrbata]

Hello @dernasherbrezon ,

The newest version of Debian is using libssl 3. The installed python3-cryptography package inside ESP-IDF is version "43.0.3".

I tried in docker with debian 12.9 and it all works for me. The cryptography package which is used in esp-idf venv is taken from cryptography pypi and it has libssl statically linked. It seems that you are using the cryptography package from debian (python3-cryptography)

# cat /etc/debian_version 
12.9

# pip show cryptography
Name: cryptography
Version: 38.0.4
Summary: cryptography is a package which provides cryptographic recipes and primitives to Python developers.
Home-page: https://github.com/pyca/cryptography
Author: The Python Cryptographic Authority and individual contributors
Author-email: [email protected]
License: BSD-3-Clause OR Apache-2.0
Location: /usr/lib/python3/dist-packages
Requires: 
Required-by: 

# find /usr/lib/python3/dist-packages/cryptography/ -type f -name '*.so'
/usr/lib/python3/dist-packages/cryptography/hazmat/bindings/_openssl.abi3.so
/usr/lib/python3/dist-packages/cryptography/hazmat/bindings/_rust.abi3.so

# ldd /usr/lib/python3/dist-packages/cryptography/hazmat/bindings/_openssl.abi3.so 
	linux-vdso.so.1 (0x00007f7bd9e3b000)
	libssl.so.3 => /lib/x86_64-linux-gnu/libssl.so.3 (0x00007f7bd9ced000)
	libcrypto.so.3 => /lib/x86_64-linux-gnu/libcrypto.so.3 (0x00007f7bd9800000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f7bd961f000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f7bd9e3d000)

# dpkg -S /usr/lib/python3/dist-packages/cryptography/hazmat/bindings/_openssl.abi3.so
python3-cryptography: /usr/lib/python3/dist-packages/cryptography/hazmat/bindings/_openssl.abi3.so

Within the properly installed and activated esp-idf environment the cryptography package should not have dependency on libssl dynamic library, because it's statically linked.

# pip show cryptography
Name: cryptography
Version: 43.0.3
Summary: cryptography is a package which provides cryptographic recipes and primitives to Python developers.
Home-page: https://github.com/pyca/cryptography
Author: The cryptography developers <[email protected]>
Author-email: The Python Cryptographic Authority and individual contributors <[email protected]>
License: Apache-2.0 OR BSD-3-Clause
Location: /root/.espressif/python_env/idf5.4_py3.11_env/lib/python3.11/site-packages
Requires: cffi
Required-by: esp_idf_nvs_partition_gen, esptool

# find /root/.espressif/python_env/idf5.4_py3.11_env/lib/python3.11/site-packages/cryptography -type f -name '*.so'
/root/.espressif/python_env/idf5.4_py3.11_env/lib/python3.11/site-packages/cryptography/hazmat/bindings/_rust.abi3.so

Have you run the install.sh and . ./export.sh scripts? How to properly install and activate esp-idf environment is described in https://docs.espressif.com/projects/esp-idf/en/v5.4/esp32/get-started/linux-macos-setup.html

Thank you and apologies if I've missed something.