Multiple .egg-info Directories in setup.py Script

[antis81]

Environment

• pip version: 20.1
• Python version: 3.7
• OS: macos 10.15

Description
After upgrading to 20.1 pip errors out with developer installation in (venv setup).
Downgrading to 20.0.2 solves it immediately so I guess something is not right with the new version.

Expected behavior

How to Reproduce

python3 -m venv venv
source venv/bin/activate
pip install --upgrade pip
pip install -e <my-source-directory>

Output

ERROR: More than one .egg-info directory found in <temp-path>

[deveshks]

Could you run the pip install commands in verbose mode using the -v flag and provide a more detailed output.

Could you also provide the package name/github link which you trying to install?

[sbidoul]

This message was introduced in #7978.

Could you also try running mkdir /tmp/test8201 && python setup.py egg_info --egg-base /tmp/test8201, and report the content of /tmp/test8201 after that (make sure it is empty before)?

[antis81]

@sbidoul Thanks for the hint!

Unfortunately I cannot provide the package information you ask for in it's full beauty. Anyway the setup.py has the purpose of generating two wheel packages (second one is an extension to the first and depends on that).

Running with egg_info --egg-base /tmp/test8201 outputs two .egg-info directories in the same base location. Schematically:

/tmp/test8201
    CppApi.egg-info
    CppApi_interactive_shell.egg-info

That explains the error message at least.

[sbidoul]

Ah, yes that explain. I never imagined a setup.py could do that.
Is that using a public setuptools plugin or something custom?

I think the previous pip version did work by accident rather than by design in that case.

[uranusjr]

Yes, IIRC the previous implementation just chose one of them arbitrarily. It probably always chooses whatever comes first in os.listdir(), but that is entirely implementation-defined and not guaranteed.

[antis81]

From the previous implementation:

filenames.sort(key=depth_of_directory)

The depth_of_directory did count path seperators -> pseudo arbitrary.

@sbidoul Yes setuptools is a public plugin.

[pradyunsg]

Yay! Thanks @antis81 for filing this issue! ^>^

If you could share the setup.py file, or a minimal reproducer, that'd be great! That'll help figure out why multiple egg-info folders are being generated, and whether that is a use case we'd want to support.

As @sbidoul mentioned, we don't know when a setup.py file ends up generating multiple egg-info directories, and whether that is correct behavior/something we'd want to support.

[antis81]

Here is a minimal setup.py script showing how multiple wheels are generated:

pip_issue_8201_setup.tar.gz

It's purpose is to split scripts into separate wheels providing a (required) fine granularity over each package's dependencies. As you can see, the packages do share some meta info -> especially the package version).

[pfmoore]

OK, so one setup.py building two projects? I'm pretty sure I'd consider that out of scope for pip.

[pradyunsg]

The file uploaded earlier, posted inline.

#!/usr/bin/env python3
#-*- coding: utf-8 -*-

import setuptools
from setuptools.command.develop import develop as SetuptoolsDevelop
from wheel.bdist_wheel import bdist_wheel
import subprocess
import wheel.bdist_wheel
import sys
import os
import shutil

"""
Setuptools based test script to build .whl python packages (wheel) for the PIP issue 8201 test package.

Usage: python3 setup.py bdist_wheel
"""


def main():
    # The actual package version ("public version identifier" in PEP 440). Should be of form x.y.z (etc.)
    public_version = "0.0.1"

    # The "local version identifier" as specified in PEP 440
    # local_version = None

    # See PEP 440 for details on supported version schemes in python packages.
    # -> https://www.python.org/dev/peps/pep-0440/
    # Essentially: they may consist of a "public version" and a "local version" component. The public version must
    # follow a somewhat simple format ("greater" comparison must be possible), so git hashes are not possible.
    # a local version is merely a label.
    version = public_version

    if '--build_package' in sys.argv:
        idx = sys.argv.index('--build_package')
        if len(sys.argv) <= idx + 1:
            print("ERROR: Require argument to --build_package")
            sys.exit(1)
        build_package = sys.argv[idx + 1]
        del sys.argv[idx]
        del sys.argv[idx]
        if build_package == 'pip-issue-8201':
            build_pip_issue_8201(version=version)
        elif build_package == 'pip-issue-8201-extensions':
            build_pip_issue_8201_extensions(version=version)
        else:
            print("ERROR: Invalid package %s" % build_package)
            sys.exit(1)
    else:
        for package in ['pip-issue-8201', 'pip-issue-8201-extensions']:
            argv = sys.argv
            if argv[0] == '-c':
                argv[0] = __file__
            args = [sys.executable] + argv + ['--build_package', package]
            p = subprocess.Popen(args)
            p.communicate()
            if p.returncode != 0:
                print("ERROR: Failed to build package %s" % package)
                sys.exit(p.returncode)
        print("Finished building packages.")


def build_issue_8201(version: str):
    # remove any previous setuptools build dir
    shutil.rmtree("build", ignore_errors=True)

    print("Starting setuptools.setup() for package pip-issue-8201")
    setuptools.setup(
        name="pip-issue-8201",
        version=version,
        maintainer="PIP Issue8201",
        maintainer_email="[email protected]",
        description="Python wheel package PIP issue 8201",
        long_description="Test package generating multiple .egg-info directories",
        url="https://github.com/pypa/pip/issues/8201",
        packages=setuptools.find_packages(".", exclude=["pip-issue-8201-extensions"]),
        classifiers=[
            'Intended Audience :: Developers',
            "License :: Open Source",
            "Programming Language :: Python :: 3",
            "Programming Language :: C++",
        ],
        keywords='pip,issue,8201',
        install_requires=["pip==20.1"]
    )
    print("Finished setuptools.setup() for package pip-issue-8201")

def build_pip_issue_8201_extensions(version: str):
    # remove previous setuptools build dir
    shutil.rmtree("build", ignore_errors=True)

    # Add a separate package for the applications, so we can easily pull in dependencies that are
    # not required for the bindings themselves (e.g. ipython).
    print("Starting setuptools.setup() for package pip-issue-8201-extensions")
    setuptools.setup(
        name="pip-issue-8201-extensions",
        version=version,
        maintainer="PIP Issue8201",
        maintainer_email="[email protected]",
        description="Python wheel package PIP issue 8201",
        long_description="Test package generating multiple .egg-info directories",
        url="https://github.com/pypa/pip/issues/8201",
        packages=setuptools.find_packages(".", exclude=["pip-issue-8201"]),
        classifiers=[
            'Intended Audience :: Developers',
            "License :: Open Source",
            "Programming Language :: Python :: 3",
            "Programming Language :: C++",
        ],
        keywords='pip,issue,8201',
        install_requires=["pip==20.1"],
        package_data={
        },

        install_requires=["pip-issue-8201==%s" % version, "pyaes", "ipython"],

        entry_points={
            # 'console_scripts': [
            # ],
            #'gui_scripts': [
            #],
        }
    )
    print("Finished setuptools.setup() for package pip-issue-8201")


if __name__ == '__main__':
    main()

[pradyunsg]

The entire design of Python packaging tooling is based on one source tree / one source distribution represents one eventually-installed package.

Using a single setup.py file, for generating 2 packages/distributions is definitely not something that'd work with these tools, and as @pfmoore noted, it's certainly out of scope for pip.

[sbidoul]

I'm closing this as something the packaging ecosystem does not support. @antis81 I'd recommend you create one subdirectory for each package each with it's own setup.py.

[vkosuri]

Hi, I am also facing the same issue with Python2.7, Is there any workaround proposed for this issue to sucessful installation?

(venv) test@virtual-machine:~$ pip --version
pip 20.1.1 from /home/test/venv/lib/python2.7/site-packages/pip (python 2.7)
(venv) test@virtual-machine:~$

[sbidoul]

@vkosuri setup.py in python-kadmin attempts to package two distributions: python-kadmin and python-kadmin-local. As explained above, this is unsupported. A possible workaround (if the package cannot be updated) is to download it manually from PyPI then run setup.py install.