Support PEP 517 (and thus poetry)

[moorepants]

poetry has a strict set of flags when invoking pip. One of those is --use-pep517. This means that poetry fails to install cyipopt using pip because PEP 517 expects pypi available build dependencies to be specified in a pyproject.toml file. PEP 517 causes difficulties in the conda build system and various hacks are needed to delete or ignore the pyproject.toml file and thus PEP 517 compliance.

See more info in:

• Reduce manual pre-installation requirements via pyproject.toml #162
• Installation of dependencies #100
• Need for a knowledgebase explanation on how to handle pyproject.toml, PEP 517, and PEP 518 conda-forge/conda-forge.github.io#1174

If we include a pyproject.toml in our source distribution then we will have to accommodate that with work arounds in our conda forge builds. If we don't include the file, then users trying to install using poetry will need a work around. We need a solution that works for both.

Here is an example error message from a poetry install command:

 => ERROR [6/8] RUN poetry add cyipopt                                                                                                               14.1s
------
 > [6/8] RUN poetry add cyipopt:
#9 0.869 Creating virtualenv app-9TtSrW0h-py3.10 in /root/.cache/pypoetry/virtualenvs
#9 1.638 Using version ^1.2.0 for cyipopt
#9 1.640
#9 1.640 Updating dependencies
#9 1.641 Resolving dependencies...
#9 5.387
#9 5.387 Writing lock file
#9 5.464
#9 5.464 Package operations: 3 installs, 0 updates, 0 removals
#9 5.464
#9 5.466   • Installing cython (0.29.32)
#9 5.467   • Installing numpy (1.23.5)
#9 11.00   • Installing cyipopt (1.2.0)
#9 13.54
#9 13.54   CalledProcessError
#9 13.54
#9 13.54   Command '['/root/.cache/pypoetry/virtualenvs/app-9TtSrW0h-py3.10/bin/python', '-m', 'pip', 'install', '--use-pep517', '--disable-pip-version-check', '--isolated', '--no-input', '--prefix', '/root/.cache/pypoetry/virtualenvs/app-9TtSrW0h-py3.10', '--no-deps', '/root/.cache/pypoetry/artifacts/a5/a8/9e/80306f8dda7dc4682a86897b5dfdc4d1c4444d5055260091388599b8ec/cyipopt-1.2.0.tar.gz']' returned non-zero exit status 1.
#9 13.54
#9 13.54   at /usr/local/lib/python3.10/subprocess.py:526 in run
#9 13.63        522│             # We don't call process.wait() as .__exit__ does that for us.
#9 13.63        523│             raise
#9 13.63        524│         retcode = process.poll()
#9 13.63        525│         if check and retcode:
#9 13.63     →  526│             raise CalledProcessError(retcode, process.args,
#9 13.63        527│                                      output=stdout, stderr=stderr)
#9 13.63        528│     return CompletedProcess(process.args, retcode, stdout, stderr)
#9 13.63        529│
#9 13.63        530│
#9 13.63
#9 13.63 The following error occurred when trying to handle this error:
#9 13.63
#9 13.63
#9 13.63   EnvCommandError
#9 13.63
#9 13.63   Command ['/root/.cache/pypoetry/virtualenvs/app-9TtSrW0h-py3.10/bin/python', '-m', 'pip', 'install', '--use-pep517', '--disable-pip-version-check', '--isolated', '--no-input', '--prefix', '/root/.cache/pypoetry/virtualenvs/app-9TtSrW0h-py3.10', '--no-deps', '/root/.cache/pypoetry/artifacts/a5/a8/9e/80306f8dda7dc4682a86897b5dfdc4d1c4444d5055260091388599b8ec/cyipopt-1.2.0.tar.gz'] errored with the following return code 1, and output:
#9 13.63   Processing /root/.cache/pypoetry/artifacts/a5/a8/9e/80306f8dda7dc4682a86897b5dfdc4d1c4444d5055260091388599b8ec/cyipopt-1.2.0.tar.gz
#9 13.63     Installing build dependencies: started
#9 13.63     Installing build dependencies: finished with status 'done'
#9 13.63     Getting requirements to build wheel: started
#9 13.63     Getting requirements to build wheel: finished with status 'error'
#9 13.63     error: subprocess-exited-with-error
#9 13.63
#9 13.63     × Getting requirements to build wheel did not run successfully.
#9 13.63     │ exit code: 1
#9 13.63     ╰─> [41 lines of output]
#9 13.63         /tmp/pip-build-env-h9al84rn/overlay/lib/python3.10/site-packages/setuptools/installer.py:27: SetuptoolsDeprecationWarning: setuptools.installer is deprecated. Requirements should be satisfied by a PEP 517 installer.
#9 13.63           warnings.warn(
#9 13.63         /root/.cache/pypoetry/virtualenvs/app-9TtSrW0h-py3.10/bin/python: No module named pip
#9 13.63         Traceback (most recent call last):
#9 13.63           File "/tmp/pip-build-env-h9al84rn/overlay/lib/python3.10/site-packages/setuptools/installer.py", line 82, in fetch_build_egg
#9 13.63             subprocess.check_call(cmd)
#9 13.63           File "/usr/local/lib/python3.10/subprocess.py", line 369, in check_call
#9 13.63             raise CalledProcessError(retcode, cmd)
#9 13.63         subprocess.CalledProcessError: Command '['/root/.cache/pypoetry/virtualenvs/app-9TtSrW0h-py3.10/bin/python', '-m', 'pip', '--disable-pip-version-check', 'wheel', '--no-deps', '-w', '/tmp/tmpxmaz4k18', '--quiet', 'numpy>=1.15']' returned non-zero exit status 1.
#9 13.63
#9 13.63         The above exception was the direct cause of the following exception:
#9 13.63
#9 13.63         Traceback (most recent call last):
#9 13.63           File "/root/.cache/pypoetry/virtualenvs/app-9TtSrW0h-py3.10/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 351, in <module>
#9 13.63             main()
#9 13.63           File "/root/.cache/pypoetry/virtualenvs/app-9TtSrW0h-py3.10/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 333, in main
#9 13.63             json_out['return_val'] = hook(**hook_input['kwargs'])
#9 13.63           File "/root/.cache/pypoetry/virtualenvs/app-9TtSrW0h-py3.10/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 118, in get_requires_for_build_wheel
#9 13.63             return hook(config_settings)
#9 13.63           File "/tmp/pip-build-env-h9al84rn/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 338, in get_requires_for_build_wheel
#9 13.63             return self._get_build_requires(config_settings, requirements=['wheel'])
#9 13.63           File "/tmp/pip-build-env-h9al84rn/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 320, in _get_build_requires
#9 13.63             self.run_setup()
#9 13.63           File "/tmp/pip-build-env-h9al84rn/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 484, in run_setup
#9 13.63             super(_BuildMetaLegacyBackend,
#9 13.63           File "/tmp/pip-build-env-h9al84rn/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 335, in run_setup
#9 13.63             exec(code, locals())
#9 13.63           File "<string>", line 26, in <module>
#9 13.63           File "/tmp/pip-build-env-h9al84rn/overlay/lib/python3.10/site-packages/setuptools/dist.py", line 874, in fetch_build_eggs
#9 13.63             resolved_dists = pkg_resources.working_set.resolve(
#9 13.63           File "/tmp/pip-build-env-h9al84rn/overlay/lib/python3.10/site-packages/pkg_resources/__init__.py", line 789, in resolve
#9 13.63             dist = best[req.key] = env.best_match(
#9 13.63           File "/tmp/pip-build-env-h9al84rn/overlay/lib/python3.10/site-packages/pkg_resources/__init__.py", line 1075, in best_match
#9 13.63             return self.obtain(req, installer)
#9 13.63           File "/tmp/pip-build-env-h9al84rn/overlay/lib/python3.10/site-packages/pkg_resources/__init__.py", line 1087, in obtain
#9 13.63             return installer(requirement)
#9 13.63           File "/tmp/pip-build-env-h9al84rn/overlay/lib/python3.10/site-packages/setuptools/dist.py", line 944, in fetch_build_egg
#9 13.63             return fetch_build_egg(self, req)
#9 13.63           File "/tmp/pip-build-env-h9al84rn/overlay/lib/python3.10/site-packages/setuptools/installer.py", line 84, in fetch_build_egg
#9 13.63             raise DistutilsError(str(e)) from e
#9 13.63         distutils.errors.DistutilsError: Command '['/root/.cache/pypoetry/virtualenvs/app-9TtSrW0h-py3.10/bin/python', '-m', 'pip', '--disable-pip-version-check', 'wheel', '--no-deps', '-w', '/tmp/tmpxmaz4k18', '--quiet', 'numpy>=1.15']' returned non-zero exit status 1.
#9 13.63         [end of output]
#9 13.63
#9 13.63     note: This error originates from a subprocess, and is likely not a problem with pip.
#9 13.63   error: subprocess-exited-with-error
#9 13.63
#9 13.63   × Getting requirements to build wheel did not run successfully.
#9 13.63   │ exit code: 1
#9 13.63   ╰─> See above for output.
#9 13.63
#9 13.63   note: This error originates from a subprocess, and is likely not a problem with pip.
#9 13.63
#9 13.63
#9 13.63   at /usr/local/lib/python3.10/site-packages/poetry/utils/env.py:1540 in _run
#9 13.71       1536│                 output = subprocess.check_output(
#9 13.71       1537│                     command, stderr=subprocess.STDOUT, env=env, **kwargs
#9 13.71       1538│                 )
#9 13.71       1539│         except CalledProcessError as e:
#9 13.71     → 1540│             raise EnvCommandError(e, input=input_)
#9 13.71       1541│
#9 13.71       1542│         return decode(output)
#9 13.71       1543│
#9 13.71       1544│     def execute(self, bin: str, *args: str, **kwargs: Any) -> int:
#9 13.71
#9 13.71 The following error occurred when trying to handle this error:
#9 13.71
#9 13.71
#9 13.71   PoetryException
#9 13.71
#9 13.71   Failed to install /root/.cache/pypoetry/artifacts/a5/a8/9e/80306f8dda7dc4682a86897b5dfdc4d1c4444d5055260091388599b8ec/cyipopt-1.2.0.tar.gz
#9 13.71
#9 13.71   at /usr/local/lib/python3.10/site-packages/poetry/utils/pip.py:58 in pip_install
#9 13.71        54│
#9 13.71        55│     try:
#9 13.71        56│         return environment.run_pip(*args)
#9 13.71        57│     except EnvCommandError as e:
#9 13.71     →  58│         raise PoetryException(f"Failed to install {path.as_posix()}") from e
#9 13.71        59│
#9 13.71
------
executor failed running [/bin/sh -c poetry add cyipopt]: exit code: 1

[moorepants]

I've been looking to at cvxpy as a model for the wheel builds (#163). It is interesting to note that the conda forge build has to remove the pyproject.toml file: https://github.com/conda-forge/cvxpy-feedstock/blob/main/recipe/meta.yaml#L12

I also noticed that pip has a --no-use-pe517 flag. I'm not sure I recall that being there before, so maybe that can simply disable pip from using a pyproject.toml file.

[moorepants]

poetry has no issues installing cyipopt if the build dependencies are installed first (which is currently the standard recommended procedure in the cyipopt docs for a pip or python setup.py install). This works on Ubuntu 22.04:

sudo apt install coinor-libipopt1v5 coinor-libipopt-dev python3-poetry python3-cachecontrol  # cachecontrol is a dep of poetry but didn't install with python3-poetry
poetry new cyipopt_install
cd cyipopt_install/
poetry add setuptools cython numpy
poetry add cyipopt
poetry run python hs071.py

[moorepants]

Some issues with the above work around noted here #201 (comment)

[moorepants]

We now include a pyproject.toml in the source tarball on pypi.