Errors encountered with fawltydeps --install-deps option #371

zz1874 · 2023-09-19T10:52:58Z

Describe the bug
When using fawltydeps with the --install-deps option in certain projects, it encounters errors, even though the package to be installed is available on PyPI.

To Reproduce

Clone the SQLAlchemy project:

git clone [email protected]:sqlalchemy/sqlalchemy.git

Under the SQLAlchemy directory, run:

fawltydeps --install-deps --json

Error Output

$ fawltydeps --install-deps --json
WARNING:fawltydeps.limited_eval:Unable to resolve Compare(left=Call(func=Attribute(value=Name(id='platform', ctx=Load()), attr='python_implementation', ctx=Load()), args=[], keywords=[]), ops=[Eq()], comparators=[Constant(value='CPython')]) from platform.python_implementation() == 'CPython' @ setup.py:22
WARNING:fawltydeps.limited_eval:Failed to parse assignment of 'IS_CPYTHON': Compare(left=Call(func=Attribute(value=Name(id='platform', ctx=Load()), attr='python_implementation', ctx=Load()), args=[], keywords=[]), ops=[Eq()], comparators=[Constant(value='CPython')]) from platform.python_implementation() == 'CPython' @ setup.py:22
WARNING:fawltydeps.limited_eval:Unable to resolve Call(func=Name(id='bool', ctx=Load()), args=[Call(func=Attribute(value=Attribute(value=Name(id='os', ctx=Load()), attr='environ', ctx=Load()), attr='get', ctx=Load()), args=[Constant(value='DISABLE_SQLALCHEMY_CEXT')], keywords=[])], keywords=[]) from bool(os.environ.get('DISABLE_SQLALCHEMY_CEXT')) @ setup.py:23
WARNING:fawltydeps.limited_eval:Failed to parse assignment of 'DISABLE_EXTENSION': Call(func=Name(id='bool', ctx=Load()), args=[Call(func=Attribute(value=Attribute(value=Name(id='os', ctx=Load()), attr='environ', ctx=Load()), attr='get', ctx=Load()), args=[Constant(value='DISABLE_SQLALCHEMY_CEXT')], keywords=[])], keywords=[]) from bool(os.environ.get('DISABLE_SQLALCHEMY_CEXT')) @ setup.py:23
WARNING:fawltydeps.limited_eval:Unable to resolve Call(func=Name(id='bool', ctx=Load()), args=[Call(func=Attribute(value=Attribute(value=Name(id='os', ctx=Load()), attr='environ', ctx=Load()), attr='get', ctx=Load()), args=[Constant(value='REQUIRE_SQLALCHEMY_CEXT')], keywords=[])], keywords=[]) from bool(os.environ.get('REQUIRE_SQLALCHEMY_CEXT')) @ setup.py:24
WARNING:fawltydeps.limited_eval:Failed to parse assignment of 'REQUIRE_EXTENSION': Call(func=Name(id='bool', ctx=Load()), args=[Call(func=Attribute(value=Attribute(value=Name(id='os', ctx=Load()), attr='environ', ctx=Load()), attr='get', ctx=Load()), args=[Constant(value='REQUIRE_SQLALCHEMY_CEXT')], keywords=[])], keywords=[]) from bool(os.environ.get('REQUIRE_SQLALCHEMY_CEXT')) @ setup.py:24
WARNING:fawltydeps.packages:Command failed: ['/tmp/tmpwpvkb2g5/bin/pip', 'install', '--no-deps', '--quiet', '--disable-pip-version-check', 'aiomysql', 'aiosqlite', 'asyncmy', 'asyncpg', 'changelog', 'cx_oracle', 'greenlet', 'mariadb', 'mypy', 'mysql-connector-python', 'mysqlclient', 'oracledb', 'pg8000', 'psycopg', 'psycopg2', 'psycopg2-binary', 'psycopg2cffi', 'pymssql', 'pymysql', 'pyodbc', 'sphinx-autobuild', 'sphinx-copybutton', 'sphinx-paramlinks', 'sqlcipher3_binary', 'zzzeeksphinx']
WARNING:fawltydeps.packages:Output:
  error: subprocess-exited-with-error
  
  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [30 lines of output]
      /bin/sh: 1: mariadb_config: not found
      Traceback (most recent call last):
        File "/tmp/tmpwpvkb2g5/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 363, in <module>
          main()
        File "/tmp/tmpwpvkb2g5/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 345, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
        File "/tmp/tmpwpvkb2g5/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 130, in get_requires_for_build_wheel
          return hook(config_settings)
        File "/tmp/pip-build-env-mrctgcqj/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 355, in get_requires_for_build_wheel
          return self._get_build_requires(config_settings, requirements=['wheel'])
        File "/tmp/pip-build-env-mrctgcqj/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 325, in _get_build_requires
          self.run_setup()
        File "/tmp/pip-build-env-mrctgcqj/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 507, in run_setup
          super(_BuildMetaLegacyBackend, self).run_setup(setup_script=setup_script)
        File "/tmp/pip-build-env-mrctgcqj/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 341, in run_setup
          exec(code, locals())
        File "<string>", line 27, in <module>
        File "/tmp/pip-install-70ktvdpy/mariadb_13243be35f6d4dcb89c090c8e43d5e93/mariadb_posix.py", line 62, in get_config
          cc_version = mariadb_config(config_prg, "cc_version")
        File "/tmp/pip-install-70ktvdpy/mariadb_13243be35f6d4dcb89c090c8e43d5e93/mariadb_posix.py", line 28, in mariadb_config
          raise EnvironmentError(
      OSError: mariadb_config not found.
      
      This error typically indicates that MariaDB Connector/C, a dependency which
      must be preinstalled, is not found.
      If MariaDB Connector/C is not installed, see installation instructions
      If MariaDB Connector/C is installed, either set the environment variable
      MARIADB_CONFIG or edit the configuration file 'site.cfg' to set the
       'mariadb_config' option to the file location of the mariadb_config utility.
      
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

WARNING:fawltydeps.packages:Failed to install 'mariadb'
WARNING:fawltydeps.packages:Output:
  error: subprocess-exited-with-error
  
  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [30 lines of output]
      /bin/sh: 1: mariadb_config: not found
      Traceback (most recent call last):
        File "/tmp/tmpwpvkb2g5/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 363, in <module>
          main()
        File "/tmp/tmpwpvkb2g5/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 345, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
        File "/tmp/tmpwpvkb2g5/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 130, in get_requires_for_build_wheel
          return hook(config_settings)
        File "/tmp/pip-build-env-qh1kq7ji/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 355, in get_requires_for_build_wheel
          return self._get_build_requires(config_settings, requirements=['wheel'])
        File "/tmp/pip-build-env-qh1kq7ji/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 325, in _get_build_requires
          self.run_setup()
        File "/tmp/pip-build-env-qh1kq7ji/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 507, in run_setup
          super(_BuildMetaLegacyBackend, self).run_setup(setup_script=setup_script)
        File "/tmp/pip-build-env-qh1kq7ji/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 341, in run_setup
          exec(code, locals())
        File "<string>", line 27, in <module>
        File "/tmp/pip-install-3805dx95/mariadb_b51eef3d8b454a9881b5e3156ff4f73f/mariadb_posix.py", line 62, in get_config
          cc_version = mariadb_config(config_prg, "cc_version")
        File "/tmp/pip-install-3805dx95/mariadb_b51eef3d8b454a9881b5e3156ff4f73f/mariadb_posix.py", line 28, in mariadb_config
          raise EnvironmentError(
      OSError: mariadb_config not found.
      
      This error typically indicates that MariaDB Connector/C, a dependency which
      must be preinstalled, is not found.
      If MariaDB Connector/C is not installed, see installation instructions
      If MariaDB Connector/C is installed, either set the environment variable
      MARIADB_CONFIG or edit the configuration file 'site.cfg' to set the
       'mariadb_config' option to the file location of the mariadb_config utility.
      
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

WARNING:fawltydeps.packages:Failed to install 'mysqlclient'
WARNING:fawltydeps.packages:Output:
  error: subprocess-exited-with-error
  
  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [24 lines of output]
      /bin/sh: 1: pkg-config: not found
      /bin/sh: 1: pkg-config: not found
      Trying pkg-config --exists mysqlclient
      Command 'pkg-config --exists mysqlclient' returned non-zero exit status 127.
      Trying pkg-config --exists mariadb
      Command 'pkg-config --exists mariadb' returned non-zero exit status 127.
      Traceback (most recent call last):
        File "/tmp/tmpwpvkb2g5/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 363, in <module>
          main()
        File "/tmp/tmpwpvkb2g5/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 345, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
        File "/tmp/tmpwpvkb2g5/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 130, in get_requires_for_build_wheel
          return hook(config_settings)
        File "/tmp/pip-build-env-u0_qbs05/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 355, in get_requires_for_build_wheel
          return self._get_build_requires(config_settings, requirements=['wheel'])
        File "/tmp/pip-build-env-u0_qbs05/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 325, in _get_build_requires
          self.run_setup()
        File "/tmp/pip-build-env-u0_qbs05/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 341, in run_setup
          exec(code, locals())
        File "<string>", line 154, in <module>
        File "<string>", line 48, in get_config_posix
        File "<string>", line 27, in find_package_name
      Exception: Can not find valid pkg-config name.
      Specify MYSQLCLIENT_CFLAGS and MYSQLCLIENT_LDFLAGS env vars manually
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

WARNING:fawltydeps.packages:Failed to install 'psycopg2'
WARNING:fawltydeps.packages:Output:
  error: subprocess-exited-with-error
  
  × Running setup.py install for psycopg2 did not run successfully.
  │ exit code: 1
  ╰─> [40 lines of output]
      running install
      /tmp/tmpwpvkb2g5/lib/python3.10/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
        warnings.warn(
      running build
      running build_py
      creating build
      creating build/lib.linux-x86_64-3.10
      creating build/lib.linux-x86_64-3.10/psycopg2
      copying lib/_ipaddress.py -> build/lib.linux-x86_64-3.10/psycopg2
      copying lib/extras.py -> build/lib.linux-x86_64-3.10/psycopg2
      copying lib/errorcodes.py -> build/lib.linux-x86_64-3.10/psycopg2
      copying lib/_range.py -> build/lib.linux-x86_64-3.10/psycopg2
      copying lib/errors.py -> build/lib.linux-x86_64-3.10/psycopg2
      copying lib/pool.py -> build/lib.linux-x86_64-3.10/psycopg2
      copying lib/sql.py -> build/lib.linux-x86_64-3.10/psycopg2
      copying lib/extensions.py -> build/lib.linux-x86_64-3.10/psycopg2
      copying lib/tz.py -> build/lib.linux-x86_64-3.10/psycopg2
      copying lib/__init__.py -> build/lib.linux-x86_64-3.10/psycopg2
      copying lib/_json.py -> build/lib.linux-x86_64-3.10/psycopg2
      running build_ext
      building 'psycopg2._psycopg' extension
      creating build/temp.linux-x86_64-3.10
      creating build/temp.linux-x86_64-3.10/psycopg
      x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DPSYCOPG_VERSION=2.9.7 (dt dec pq3 ext lo64) -DPSYCOPG_DEBUG=1 -DPG_VERSION_NUM=150003 -DHAVE_LO64=1 -DPSYCOPG_DEBUG=1 -I/tmp/tmpwpvkb2g5/include -I/usr/include/python3.10 -I. -I/usr/include/postgresql -I/usr/include/postgresql/15/server -I/usr/include/libxml2 -c psycopg/adapter_asis.c -o build/temp.linux-x86_64-3.10/psycopg/adapter_asis.o -Wdeclaration-after-statement
      In file included from psycopg/adapter_asis.c:28:
      ./psycopg/psycopg.h:36:10: fatal error: libpq-fe.h: No such file or directory
         36 | #include <libpq-fe.h>
            |          ^~~~~~~~~~~~
      compilation terminated.
      
      It appears you are missing some prerequisite to build the package from source.
      
      You may install a binary package by installing 'psycopg2-binary' from PyPI.
      If you want to install psycopg2 from source, please install the packages
      required for the build and try again.
      
      For further information please check the 'doc/src/install.rst' file (also at
      <https://www.psycopg.org/docs/install.html>).
      
      error: command '/usr/bin/x86_64-linux-gnu-gcc' failed with exit code 1
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> psycopg2

note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure.

WARNING:fawltydeps.packages:Failed to install 'psycopg2cffi'
WARNING:fawltydeps.packages:Output:
  error: subprocess-exited-with-error
  
  × Running setup.py install for psycopg2cffi did not run successfully.
  │ exit code: 1
  ╰─> [54 lines of output]
      /tmp/tmpwpvkb2g5/lib/python3.10/site-packages/setuptools/installer.py:27: SetuptoolsDeprecationWarning: setuptools.installer is deprecated. Requirements should be satisfied by a PEP 517 installer.
        warnings.warn(
      WARNING: The wheel package is not available.
      WARNING: The wheel package is not available.
      running install
      /tmp/tmpwpvkb2g5/lib/python3.10/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
        warnings.warn(
      running build
      running build_py
      creating build
      creating build/lib.linux-x86_64-3.10
      creating build/lib.linux-x86_64-3.10/psycopg2cffi
      copying psycopg2cffi/extras.py -> build/lib.linux-x86_64-3.10/psycopg2cffi
      copying psycopg2cffi/errorcodes.py -> build/lib.linux-x86_64-3.10/psycopg2cffi
      copying psycopg2cffi/_range.py -> build/lib.linux-x86_64-3.10/psycopg2cffi
      copying psycopg2cffi/pool.py -> build/lib.linux-x86_64-3.10/psycopg2cffi
      copying psycopg2cffi/extensions.py -> build/lib.linux-x86_64-3.10/psycopg2cffi
      copying psycopg2cffi/compat.py -> build/lib.linux-x86_64-3.10/psycopg2cffi
      copying psycopg2cffi/tz.py -> build/lib.linux-x86_64-3.10/psycopg2cffi
      copying psycopg2cffi/__init__.py -> build/lib.linux-x86_64-3.10/psycopg2cffi
      copying psycopg2cffi/_json.py -> build/lib.linux-x86_64-3.10/psycopg2cffi
      copying psycopg2cffi/psycopg1.py -> build/lib.linux-x86_64-3.10/psycopg2cffi
      creating build/lib.linux-x86_64-3.10/psycopg2cffi/_impl
      copying psycopg2cffi/_impl/exceptions.py -> build/lib.linux-x86_64-3.10/psycopg2cffi/_impl
      copying psycopg2cffi/_impl/connection.py -> build/lib.linux-x86_64-3.10/psycopg2cffi/_impl
      copying psycopg2cffi/_impl/lobject.py -> build/lib.linux-x86_64-3.10/psycopg2cffi/_impl
      copying psycopg2cffi/_impl/consts.py -> build/lib.linux-x86_64-3.10/psycopg2cffi/_impl
      copying psycopg2cffi/_impl/encodings.py -> build/lib.linux-x86_64-3.10/psycopg2cffi/_impl
      copying psycopg2cffi/_impl/adapters.py -> build/lib.linux-x86_64-3.10/psycopg2cffi/_impl
      copying psycopg2cffi/_impl/libpq.py -> build/lib.linux-x86_64-3.10/psycopg2cffi/_impl
      copying psycopg2cffi/_impl/notify.py -> build/lib.linux-x86_64-3.10/psycopg2cffi/_impl
      copying psycopg2cffi/_impl/cursor.py -> build/lib.linux-x86_64-3.10/psycopg2cffi/_impl
      copying psycopg2cffi/_impl/__init__.py -> build/lib.linux-x86_64-3.10/psycopg2cffi/_impl
      copying psycopg2cffi/_impl/typecasts.py -> build/lib.linux-x86_64-3.10/psycopg2cffi/_impl
      copying psycopg2cffi/_impl/_build_libpq.py -> build/lib.linux-x86_64-3.10/psycopg2cffi/_impl
      copying psycopg2cffi/_impl/util.py -> build/lib.linux-x86_64-3.10/psycopg2cffi/_impl
      copying psycopg2cffi/_impl/xid.py -> build/lib.linux-x86_64-3.10/psycopg2cffi/_impl
      creating build/lib.linux-x86_64-3.10/psycopg2cffi/tests
      copying psycopg2cffi/tests/test_extensions.py -> build/lib.linux-x86_64-3.10/psycopg2cffi/tests
      copying psycopg2cffi/tests/test_errorcodes.py -> build/lib.linux-x86_64-3.10/psycopg2cffi/tests
      copying psycopg2cffi/tests/test_notify.py -> build/lib.linux-x86_64-3.10/psycopg2cffi/tests
      copying psycopg2cffi/tests/__init__.py -> build/lib.linux-x86_64-3.10/psycopg2cffi/tests
      running build_ext
      generating cffi module 'build/temp.linux-x86_64-3.10/psycopg2cffi._impl._libpq.c'
      creating build/temp.linux-x86_64-3.10
      building 'psycopg2cffi._impl._libpq' extension
      creating build/temp.linux-x86_64-3.10/build
      creating build/temp.linux-x86_64-3.10/build/temp.linux-x86_64-3.10
      x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.10 -I/usr/include/postgresql -I/tmp/tmpwpvkb2g5/include -I/usr/include/python3.10 -c build/temp.linux-x86_64-3.10/psycopg2cffi._impl._libpq.c -o build/temp.linux-x86_64-3.10/build/temp.linux-x86_64-3.10/psycopg2cffi._impl._libpq.o
      build/temp.linux-x86_64-3.10/psycopg2cffi._impl._libpq.c:576:10: fatal error: postgres_ext.h: No such file or directory
        576 | #include <postgres_ext.h>
            |          ^~~~~~~~~~~~~~~~
      compilation terminated.
      error: command '/usr/bin/x86_64-linux-gnu-gcc' failed with exit code 1
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> psycopg2cffi

note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure.

ERROR:fawltydeps.main:Unresolved dependencies: mariadb, mysqlclient, psycopg2, psycopg2cffi
FawltyDeps is unable to find the above packages with the configured package resolvers. Consider using --pyenv if these packages are already installed somewhere, or --custom-mapping-file to take full control of the package-to-import-names mapping.

Expected behavior
FawltyDeps should correctly detect undeclared and unused dependencies.

Environment

OS name + version: Ubuntu 22.04.2 LTS
Version of the code: v0.13.1

Additional context
There's no additional context.

The text was updated successfully, but these errors were encountered:

jherland · 2023-09-19T11:13:47Z

So it seems these dependencies fail because they rely on other, non-Python (possibly system-level) dependencies to be installed, and these simply cannot be handled via pip install (at least not without more user intervention).

I'm not sure if FD really can do a better job in this case? It's either up to the user running FD (and who is hopefully more familiar with the project being tested) to actually provide these deps, or to use custom mapping and/or --ignore-undeclared to step around this problem. Any better ideas?

Not sure how this affects our experiment? Maybe start building a custom-mapping of known dependencies that need tight system integration, and make a note of this? I'm sure this will affect quite a number of packages from PyPI.

Nour-Mws · 2023-09-19T13:00:38Z

Not sure how this affects our experiment? Maybe start building a custom-mapping of known dependencies that need tight system integration, and make a note of this? I'm sure this will affect quite a number of packages from PyPI.

That's an excellent start point. I would second this option. We'd also have much more control and visibility than with, say, an environment where the packages are pre-installed.

zz1874 · 2023-09-19T14:57:34Z

Thanks for your inputs! For our experiment, creating a custom mapping of dependencies that require tight system integration and noting this is a great idea, I'll move forward with it 💪

zz1874 added the type: bug label Sep 19, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Errors encountered with fawltydeps --install-deps option #371

Errors encountered with fawltydeps --install-deps option #371

zz1874 commented Sep 19, 2023

jherland commented Sep 19, 2023

Nour-Mws commented Sep 19, 2023

zz1874 commented Sep 19, 2023

Errors encountered with fawltydeps --install-deps option #371

Errors encountered with fawltydeps --install-deps option #371

Comments

zz1874 commented Sep 19, 2023

jherland commented Sep 19, 2023

Nour-Mws commented Sep 19, 2023

zz1874 commented Sep 19, 2023