add-ons installed as editable are not found anymore

[ksuess]

BUG/PROBLEM REPORT (OR OTHER COMMON ISSUE)

What I did:

• New project with cookieplone
• backend: make build
• backend: make start

What I expect to happen:

What actually happened:

ModuleNotFoundError: No module named 'project_title'

What version of Plone/ Addons I am using:

Python 3.12
Plone 6.1, Plone 6.0

katjasuss@purpur backend % make start
+ which python3
+ python3 -c 'import sys; print((int(sys.version_info[0]), int(sys.version_info[1])) >= tuple(map(int, '\''3.10'\''.split('\''.'\''))))'
+ dirname /Users/katjasuss/Desktop/_temp/scratch_training_2025_installation/project-title/backend/Makefile
+ dirname /Users/katjasuss/Desktop/_temp/scratch_training_2025_installation/project-title/backend/Makefile
+ dirname /Users/katjasuss/Desktop/_temp/scratch_training_2025_installation/project-title/backend/Makefile
+ dirname /Users/katjasuss/Desktop/_temp/scratch_training_2025_installation/project-title/backend/Makefile
+ dirname /Users/katjasuss/Desktop/_temp/scratch_training_2025_installation/project-title/backend/Makefile
+ dirname /Users/katjasuss/Desktop/_temp/scratch_training_2025_installation/project-title/backend/Makefile
+ dirname /Users/katjasuss/Desktop/_temp/scratch_training_2025_installation/project-title/backend/Makefile
+ dirname /Users/katjasuss/Desktop/_temp/scratch_training_2025_installation/project-title/backend/Makefile
+ dirname /Users/katjasuss/Desktop/_temp/scratch_training_2025_installation/project-title/backend/Makefile
+ dirname /Users/katjasuss/Desktop/_temp/scratch_training_2025_installation/project-title/backend/Makefile
+ dirname /Users/katjasuss/Desktop/_temp/scratch_training_2025_installation/project-title/backend/Makefile
+ dirname /Users/katjasuss/Desktop/_temp/scratch_training_2025_installation/project-title/backend/Makefile
+ dirname /Users/katjasuss/Desktop/_temp/scratch_training_2025_installation/project-title/backend/Makefile
+ dirname /Users/katjasuss/Desktop/_temp/scratch_training_2025_installation/project-title/backend/Makefile
+ dirname /Users/katjasuss/Desktop/_temp/scratch_training_2025_installation/project-title/backend/Makefile
+ PYTHONWARNINGS=ignore
+ /Users/katjasuss/Desktop/_temp/scratch_training_2025_installation/project-title/backend/.venv/bin/runwsgi instance/etc/zope.ini
2025-02-26 15:17:00,566 INFO    [chameleon.config:39][MainThread] directory cache: /Users/katjasuss/Desktop/_temp/scratch_training_2025_installation/project-title/backend/instance/var/cache.
2025-02-26 15:17:12,251 INFO    [ZODB.blob:349][MainThread] (55307) Blob directory /Users/katjasuss/Desktop/_temp/scratch_training_2025_installation/project-title/backend/instance/var/blobs does not exist. Selected `bushy` layout. 
2025-02-26 15:17:12,252 INFO    [ZODB.blob:349][MainThread] (55307) Blob directory '/Users/katjasuss/Desktop/_temp/scratch_training_2025_installation/project-title/backend/instance/var/blobs/' does not exist. Created new directory.
2025-02-26 15:17:12,252 INFO    [ZODB.blob:349][MainThread] (55307) Blob temporary directory '/Users/katjasuss/Desktop/_temp/scratch_training_2025_installation/project-title/backend/instance/var/blobs/tmp' does not exist. Created new directory.
2025-02-26 15:17:13,067 INFO    [plone.restapi.patches:16][MainThread] PATCH: Disabled ZPublisher.HTTPRequest.ZopeFieldStorage.VALUE_LIMIT. This enables file uploads larger than 1MB.
2025-02-26 15:17:13,266 ERROR   [plone.autoinclude.loader:60][MainThread] Could not import project_title. Set environment variable AUTOINCLUDE_ALLOW_MODULE_NOT_FOUND_ERROR=1 if you want to allow this. Or set it to 'project_title' to only allow for this project. Can be a comma-separated list of project names. Or replace the z3c.autoinclude.plugin entry point of this project with plone.autoinclude.plugin and a module name.
Traceback (most recent call last):
  File "/Users/katjasuss/Desktop/_temp/scratch_training_2025_installation/project-title/backend/.venv/lib/python3.12/site-packages/zope/configuration/xmlconfig.py", line 448, in endElementNS
    self.context.end()
  File "/Users/katjasuss/Desktop/_temp/scratch_training_2025_installation/project-title/backend/.venv/lib/python3.12/site-packages/zope/configuration/config.py", line 748, in end
    self.stack.pop().finish()
  File "/Users/katjasuss/Desktop/_temp/scratch_training_2025_installation/project-title/backend/.venv/lib/python3.12/site-packages/zope/configuration/config.py", line 918, in finish
    actions = self.handler(context, **args)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/katjasuss/Desktop/_temp/scratch_training_2025_installation/project-title/backend/.venv/lib/python3.12/site-packages/plone/autoinclude/zcml.py", line 37, in includePluginsDirective
    dists = loader.load_packages(target)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/katjasuss/Desktop/_temp/scratch_training_2025_installation/project-title/backend/.venv/lib/python3.12/site-packages/plone/autoinclude/loader.py", line 139, in load_packages
    z3c_dists = load_z3c_packages(target=target)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/katjasuss/Desktop/_temp/scratch_training_2025_installation/project-title/backend/.venv/lib/python3.12/site-packages/plone/autoinclude/loader.py", line 50, in load_z3c_packages
    dist = importlib.import_module(module_name)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/katjasuss/.pyenv/versions/3.12.2/lib/python3.12/importlib/__init__.py", line 90, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1324, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'project_title'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/katjasuss/Desktop/_temp/scratch_training_2025_installation/project-title/backend/.venv/bin/runwsgi", line 10, in <module>
    sys.exit(main())
             ^^^^^^
  File "/Users/katjasuss/Desktop/_temp/scratch_training_2025_installation/project-title/backend/.venv/lib/python3.12/site-packages/Zope2/Startup/serve.py", line 251, in main
    return command.run()
           ^^^^^^^^^^^^^
  File "/Users/katjasuss/Desktop/_temp/scratch_training_2025_installation/project-title/backend/.venv/lib/python3.12/site-packages/Zope2/Startup/serve.py", line 189, in run
    app = self.loadapp(app_spec, name=app_name, relative_to=base,
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/katjasuss/Desktop/_temp/scratch_training_2025_installation/project-title/backend/.venv/lib/python3.12/site-packages/Zope2/Startup/serve.py", line 220, in loadapp
    return loadapp(app_spec, name=name, relative_to=relative_to, **kw)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/katjasuss/Desktop/_temp/scratch_training_2025_installation/project-title/backend/.venv/lib/python3.12/site-packages/paste/deploy/loadwsgi.py", line 246, in loadapp
    return loadobj(APP, uri, name=name, **kw)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/katjasuss/Desktop/_temp/scratch_training_2025_installation/project-title/backend/.venv/lib/python3.12/site-packages/paste/deploy/loadwsgi.py", line 271, in loadobj
    return context.create()
           ^^^^^^^^^^^^^^^^
  File "/Users/katjasuss/Desktop/_temp/scratch_training_2025_installation/project-title/backend/.venv/lib/python3.12/site-packages/paste/deploy/loadwsgi.py", line 738, in create
    return self.object_type.invoke(self)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/katjasuss/Desktop/_temp/scratch_training_2025_installation/project-title/backend/.venv/lib/python3.12/site-packages/paste/deploy/loadwsgi.py", line 198, in invoke
    app = context.app_context.create()
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/katjasuss/Desktop/_temp/scratch_training_2025_installation/project-title/backend/.venv/lib/python3.12/site-packages/paste/deploy/loadwsgi.py", line 738, in create
    return self.object_type.invoke(self)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/katjasuss/Desktop/_temp/scratch_training_2025_installation/project-title/backend/.venv/lib/python3.12/site-packages/paste/deploy/loadwsgi.py", line 136, in invoke
    return fix_call(context.object, context.global_conf, **context.local_conf)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/katjasuss/Desktop/_temp/scratch_training_2025_installation/project-title/backend/.venv/lib/python3.12/site-packages/paste/deploy/util.py", line 61, in fix_call
    val = callable(*args, **kw)
          ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/katjasuss/Desktop/_temp/scratch_training_2025_installation/project-title/backend/.venv/lib/python3.12/site-packages/Zope2/Startup/run.py", line 61, in make_wsgi_app
    starter.prepare()
  File "/Users/katjasuss/Desktop/_temp/scratch_training_2025_installation/project-title/backend/.venv/lib/python3.12/site-packages/Zope2/Startup/starter.py", line 38, in prepare
    self.startZope()
  File "/Users/katjasuss/Desktop/_temp/scratch_training_2025_installation/project-title/backend/.venv/lib/python3.12/site-packages/Zope2/Startup/starter.py", line 94, in startZope
    Zope2.startup_wsgi()
  File "/Users/katjasuss/Desktop/_temp/scratch_training_2025_installation/project-title/backend/.venv/lib/python3.12/site-packages/Zope2/__init__.py", line 36, in startup_wsgi
    _startup()
  File "/Users/katjasuss/Desktop/_temp/scratch_training_2025_installation/project-title/backend/.venv/lib/python3.12/site-packages/Zope2/App/startup.py", line 126, in startup
    load_zcml()
  File "/Users/katjasuss/Desktop/_temp/scratch_training_2025_installation/project-title/backend/.venv/lib/python3.12/site-packages/Zope2/App/startup.py", line 41, in load_zcml
    load_site()
  File "/Users/katjasuss/Desktop/_temp/scratch_training_2025_installation/project-title/backend/.venv/lib/python3.12/site-packages/Zope2/App/zcml.py", line 45, in load_site
    _context = xmlconfig.file(site_zcml)
               ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/katjasuss/Desktop/_temp/scratch_training_2025_installation/project-title/backend/.venv/lib/python3.12/site-packages/zope/configuration/xmlconfig.py", line 717, in file
    include(context, name, package)
  File "/Users/katjasuss/Desktop/_temp/scratch_training_2025_installation/project-title/backend/.venv/lib/python3.12/site-packages/zope/configuration/xmlconfig.py", line 614, in include
    processxmlfile(f, context)
  File "/Users/katjasuss/Desktop/_temp/scratch_training_2025_installation/project-title/backend/.venv/lib/python3.12/site-packages/zope/configuration/xmlconfig.py", line 464, in processxmlfile
    parser.parse(src)
  File "/Users/katjasuss/.pyenv/versions/3.12.2/lib/python3.12/xml/sax/expatreader.py", line 105, in parse
    xmlreader.IncrementalParser.parse(self, source)
  File "/Users/katjasuss/.pyenv/versions/3.12.2/lib/python3.12/xml/sax/xmlreader.py", line 124, in parse
    self.feed(buffer)
  File "/Users/katjasuss/.pyenv/versions/3.12.2/lib/python3.12/xml/sax/expatreader.py", line 211, in feed
    self._parser.Parse(data, isFinal)
  File "./Modules/pyexpat.c", line 470, in EndElement
  File "/Users/katjasuss/.pyenv/versions/3.12.2/lib/python3.12/xml/sax/expatreader.py", line 375, in end_element_ns
    self._cont_handler.endElementNS(pair, None)
  File "/Users/katjasuss/Desktop/_temp/scratch_training_2025_installation/project-title/backend/.venv/lib/python3.12/site-packages/zope/configuration/xmlconfig.py", line 450, in endElementNS
    self._handle_exception(ex, info)
  File "/Users/katjasuss/Desktop/_temp/scratch_training_2025_installation/project-title/backend/.venv/lib/python3.12/site-packages/zope/configuration/xmlconfig.py", line 448, in endElementNS
    self.context.end()
  File "/Users/katjasuss/Desktop/_temp/scratch_training_2025_installation/project-title/backend/.venv/lib/python3.12/site-packages/zope/configuration/config.py", line 748, in end
    self.stack.pop().finish()
  File "/Users/katjasuss/Desktop/_temp/scratch_training_2025_installation/project-title/backend/.venv/lib/python3.12/site-packages/zope/configuration/config.py", line 918, in finish
    actions = self.handler(context, **args)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/katjasuss/Desktop/_temp/scratch_training_2025_installation/project-title/backend/.venv/lib/python3.12/site-packages/OFS/metaconfigure.py", line 47, in loadProducts
    xmlconfig.include(_context, zcml, package=product)
  File "/Users/katjasuss/Desktop/_temp/scratch_training_2025_installation/project-title/backend/.venv/lib/python3.12/site-packages/zope/configuration/xmlconfig.py", line 614, in include
    processxmlfile(f, context)
  File "/Users/katjasuss/Desktop/_temp/scratch_training_2025_installation/project-title/backend/.venv/lib/python3.12/site-packages/zope/configuration/xmlconfig.py", line 464, in processxmlfile
    parser.parse(src)
  File "/Users/katjasuss/.pyenv/versions/3.12.2/lib/python3.12/xml/sax/expatreader.py", line 105, in parse
    xmlreader.IncrementalParser.parse(self, source)
  File "/Users/katjasuss/.pyenv/versions/3.12.2/lib/python3.12/xml/sax/xmlreader.py", line 124, in parse
    self.feed(buffer)
  File "/Users/katjasuss/.pyenv/versions/3.12.2/lib/python3.12/xml/sax/expatreader.py", line 211, in feed
    self._parser.Parse(data, isFinal)
  File "./Modules/pyexpat.c", line 470, in EndElement
  File "/Users/katjasuss/.pyenv/versions/3.12.2/lib/python3.12/xml/sax/expatreader.py", line 375, in end_element_ns
    self._cont_handler.endElementNS(pair, None)
  File "/Users/katjasuss/Desktop/_temp/scratch_training_2025_installation/project-title/backend/.venv/lib/python3.12/site-packages/zope/configuration/xmlconfig.py", line 450, in endElementNS
    self._handle_exception(ex, info)
  File "/Users/katjasuss/Desktop/_temp/scratch_training_2025_installation/project-title/backend/.venv/lib/python3.12/site-packages/zope/configuration/xmlconfig.py", line 237, in _handle_exception
    raise ZopeXMLConfigurationError(info, ex)
zope.configuration.xmlconfig.ZopeXMLConfigurationError: File "/Users/katjasuss/Desktop/_temp/scratch_training_2025_installation/project-title/backend/.venv/lib/python3.12/site-packages/Products/CMFPlone/meta.zcml", line 67.2-71.8
    File "/Users/katjasuss/Desktop/_temp/scratch_training_2025_installation/project-title/backend/instance/etc/site.zcml", line 9.2-9.39
    ModuleNotFoundError: No module named 'project_title'

[ksuess]

Problem has been originally reported on training setup
plone/training#925

[stevepiercy]

Related: #4126.

This is hitting Plone hard.

• Problem fetching a module: plone.autoinclude#25
• [BUG] packages of the form package.something renamed to package_something in 75.8.1 pypa/setuptools#4853 (comment)

Basically we need to stop using pkg_resources yesterday.

https://setuptools.pypa.io/en/latest/pkg_resources.html

[mauritsvanrees]

The main technical issue is in plone.autoinclude, which Steve already linked above.

I have first fixed the test setup today, as GitHub Actions had already been failing since at least earlier this year. See plone/plone.autoinclude#27 which has been merged. Warning: this drops support for Plone 5.2 and for Python 3.8. Also, PyPy and Buildout 2 and 3 are not tested anymore, though they might still work.

And then we need an actual fix of the problem. There are currently three candidate PRs:

• Work around breaking install plone.autoinclude#26
• Fix an error when loading namespace packages plone.autoinclude#28
• Fix issue 25: project versus module name plone.autoinclude#30

That last one is mine, and it is the only green one. So I recommend it. :-)

Meanwhile, I mention some workarounds on community.plone.org. Some other workarounds are above it.

Lastly: yes, we should replace all usage of pkg_resources. I added #4126 for this yesterday.

[davisagli]

Notes from my study of this problem today:

setuptools 75.8.1 was released yesterday. It changed the filename that is produced when an sdist is built into a wheel to match the standard from PEP 491. Now a period in the package name is turned into an underscore in the distribution name. This caused at least two problems:

A. plone.autoinclude tries to find a package matching the name of the distribution where the plone entrypoint was declared. This broke since they no longer match.
B. pkg_resources.get_distribution(name) with a name that includes periods no longer works.

Important note 1: this happens when building an sdist into a wheel even if setuptools is pinned like we normally do, because the build_wheel process runs in an isolated environment that installs setuptools without the pin. (A workaround for this is to add requires = ["setuptools<=75.8.0"] to the [build-system] section of pyproject.toml, though of course that's not possible when building an sdist that was already released without this.)

Important note 2: Reproducing the problem locally can be complicated because if pip finds an existing wheel in its cache with the correct version, it will use it instead of building a new one. To avoid this, purge the cache with pip cache purge.

Problem B seems to have been resolved at least for now in setuptools 75.8.2 which was released today. So we don't need to stop using pkg_resources today, but it's a good reminder that it is on its way out and we should move away from it sooner rather than later.

I tested @mauritsvanrees' solution for Problem A in plone/plone.autoinclude#30 with my project and it seems to fix the problem. We'll need a new release of plone.autoinclude with the fix, and will have to show people how to use it in older Plone releases.

[mauritsvanrees]

I have published plone.autoinclude 2.0.0 with a fix for the auto-include zcml problems.

Warning: this plone.autoinclude version requires Python 3.9 or higher. Older Python versions are out of security support anyway.

[ksuess]

Thank you @mauritsvanrees !

I can confirm that the new version of plone.autoinclude fixes the problem,
for example in training set up of mastering Plone development, plone/training#925