PLIP Replace pkg_resources in normal code

[mauritsvanrees]

PLIP (Plone Improvement Proposal)

We already want to get rid of pkg_resources style namespaces, see issue #3928.

But pkg_resources in its entirety is basically deprecated, so we should stop calling it in "normal" Python code as well. In fact, some code already does not work anymore, but it depends on how you have installed Plone. See several comments by me starting here in a PR on CMFPlone.

Technically, this does not need to be a PLIP, and we do not need explicit improvement from any team, and this can all be done in a bugfix release of Plone. But I wanted to make it more visible and more easily findable. And this affects lots of packages.

Responsible Persons

Proposer: Maurits van Rees (@mauritsvanrees)

Seconder: Steve Piercy (@stevepiercy), Gil Forcada Codinachs (@gforcada)

Abstract

Go through all code used in Plone 6, including in Zope code. Main target is branches that are used in Plone 6.1 and 6.2. If we have time, fixing it in 6.0 would be welcome as well.

Motivation

Several uses of pkg_resources actually do not work anymore in Python 3.13. This was kind-of known, but I did not really notice any problems until now. But here it is.

In a checkout of buildout.coredev 6.2, start a Zope instance based on pip, so using the Makefile.

Very important: use Python 3.13. For example with export PRIMARY_PYTHON=python3.13. Or otherwise make sure that python3 resolves to python3.13.

Maybe first call make install twice, just to be sure, there is some stuff still to fix there. But technically, just one make command should be enough:

$ make zope-start
...
Traceback (most recent call last):
  File "/Users/maurits/community/plone-coredev/6.2/.venv/lib/python3.13/site-packages/zope/configuration/xmlconfig.py", line 448, in endElementNS
    self.context.end()
    ~~~~~~~~~~~~~~~~^^
  File "/Users/maurits/community/plone-coredev/6.2/.venv/lib/python3.13/site-packages/zope/configuration/config.py", line 748, in end
    self.stack.pop().finish()
    ~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/Users/maurits/community/plone-coredev/6.2/.venv/lib/python3.13/site-packages/zope/configuration/config.py", line 918, in finish
    actions = self.handler(context, **args)
  File "/Users/maurits/community/plone-coredev/6.2/.venv/lib/python3.13/site-packages/plone/autoinclude/zcml.py", line 37, in includePluginsDirective
    dists = loader.load_packages(target)
  File "/Users/maurits/community/plone-coredev/6.2/.venv/lib/python3.13/site-packages/plone/autoinclude/loader.py", line 139, in load_packages
    z3c_dists = load_z3c_packages(target=target)
  File "/Users/maurits/community/plone-coredev/6.2/.venv/lib/python3.13/site-packages/plone/autoinclude/loader.py", line 50, in load_z3c_packages
    dist = importlib.import_module(module_name)
  File "/Users/maurits/.pyenv/versions/3.13.1/lib/python3.13/importlib/__init__.py", line 88, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 1026, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "/Users/maurits/community/plone-coredev/6.2/src/plone.app.upgrade/plone/app/upgrade/__init__.py", line 1, in <module>
    from plone.app.upgrade.utils import alias_module
  File "/Users/maurits/community/plone-coredev/6.2/src/plone.app.upgrade/plone/app/upgrade/utils.py", line 28, in <module>
    plone_version = pkg_resources.get_distribution("Products.CMFPlone").version
                    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^
  File "/Users/maurits/community/plone-coredev/6.2/.venv/lib/python3.13/site-packages/pkg_resources/__init__.py", line 529, in get_distribution
    dist = get_provider(dist)
  File "/Users/maurits/community/plone-coredev/6.2/.venv/lib/python3.13/site-packages/pkg_resources/__init__.py", line 412, in get_provider
    return working_set.find(moduleOrReq) or require(str(moduleOrReq))[0]
                                            ~~~~~~~^^^^^^^^^^^^^^^^^^
  File "/Users/maurits/community/plone-coredev/6.2/.venv/lib/python3.13/site-packages/pkg_resources/__init__.py", line 1065, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/Users/maurits/community/plone-coredev/6.2/.venv/lib/python3.13/site-packages/pkg_resources/__init__.py", line 892, in resolve
    dist = self._resolve_dist(
        req, best, replace_conflicting, env, installer, required_by, to_activate
    )
  File "/Users/maurits/community/plone-coredev/6.2/.venv/lib/python3.13/site-packages/pkg_resources/__init__.py", line 933, in _resolve_dist
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'Products.CMFPlone' distribution was not found and is required by the application

What is happening here? With Python 3.13, in a pip-only install, and Products.CMFPlone in editable install, you cannot even start a Zope instance.

Similarly, running tests then also fails, see my comments that I link above.

BTW, good old Buildout is running fine, wondering what the fuzz is about, though it has its own problems. ;-)

There also seems to be a problem with entry points, probably caused by the setuptools change, resulting in an error during install:

error msg: TypeError: ('Expected str, Requirement, or Distribution', None)

See for example plone/buildout.coredev#990

Assumptions

We want to be able to run Plone with Python 3.13 in a pip-only environment, so without Buildout, and use editable install for some packages.

Proposal & Implementation

Go through all used packages and replace any usage of pkg_resources with something else. Ignore the namespaces, that is a different topic.

Some links:

• pkg_resources documentation, which says at the top: "Use of pkg_resources is deprecated in favor of importlib.resources, importlib.metadata and their backports (importlib_resources, importlib_metadata). Some useful APIs are also provided by packaging (e.g. requirements and version parsing). Users should refrain from new usage of pkg_resources and should work to port to importlib-based solutions."
• Migration guide from pkg_resources to import lib.metadata
• Migration guide from pkg_resources to importlib.resources.

Note that we don't need the back ports packages (those with underscores), but we can use the standard lib: importlib.metadata and importlib.resources.

Let's gather some code samples here:

- __version__ = pkg_resources.require("Products.CMFPlone")[0].version
+ from importlib.metadata import version
+ __version__ = version("Products.CMFPlone")

Another one:

+from importlib.metadata import distribution
+from importlib.metadata import PackageNotFoundError
-import pkg_resources

 try:
-    pkg_resources.get_distribution("Products.CMFPlacefulWorkflow")
+    distribution("Products.CMFPlacefulWorkflow")
-except pkg_resources.DistributionNotFound:
+except PackageNotFoundError:
    ...

And files as well:

+from importlib.resources import files
-import pkg_resources

-TEST_IMAGE = pkg_resources.resource_filename("plone.app.caching.tests", "test.gif")
+TEST_IMAGE = str(files("plone.app.caching") / "tests" / "test.gif")

Tip

Note that importlib.resources.files returns a pathlib.Path, but most of the times our code expects a string, that's why it is converted to it already

Version parsing:

-from pkg_resources import parse_version
+from packaging import version
 
-HAS_PLONE5 = parse_version(env.plone_version()) >= parse_version("5.0b2")
+HAS_PLONE5 = version.parse(env.plone_version()) >= version.parse("5.0b2")

Deliverables

Grepping in 6.2 code, I see at least these:

• Products.CMFCore Drop pkg_resources usage zopefoundation/Products.CMFCore#146
• Products.CMFPlone Replace pkg_resources usage #4129
• Products.GenericSetup Replace pkg_resources with importlib.metadata and packaging. zopefoundation/Products.GenericSetup#136
• Zope Drop pkg_resources usage zopefoundation/Zope#1253
• collective.monkeypatcher Config with plone.meta and drop pkg_resources collective.monkeypatcher#15
• diazo Replace pkg_resources diazo#90
• five.intid Replace pkg_resources five.intid#35
• paste Remove pkg_resources usage pasteorg/paste#105
• plone.api Replace pkg_resources plone.api#567
• plone.app.caching Replace pkg_resources plone.app.caching#147
• plone.app.contentmenu Replace pkg_resources plone.app.contentmenu#75
• plone.app.contenttypes Replace pkg_resources plone.app.contenttypes#719
• plone.app.dexterity Replace pkg_resources plone.app.dexterity#403
• plone.app.layout Drop pkg_resources usage plone.app.layout#394
• plone.app.multilingual Drop pkg_resources usage plone.app.multilingual#485
• plone.app.relationfield Drop pkg_resources usage plone.app.relationfield#62
• plone.app.robotframework Drop pkg_resources usage plone.app.robotframework#169
• plone.app.theming Drop pkg_resources usage plone.app.theming#251
• plone.app.upgrade Drop pkg_resources usage plone.app.upgrade#337
• plone.app.users Drop pkg_resources usage plone.app.users#137
• plone.autoinclude Replace pkg_resources plone.autoinclude#31
• plone.distribution Drop pkg_resources usage plone.distribution#112
• plone.restapi Drop pkg_resources usage plone.restapi#1882
• plone.schemaeditor Drop pkg_resources usage plone.schemaeditor#132
• plone.testing Drop pkg resources usage plone.testing#106
• plone.volto Drop pkg_resources usage plone.volto#188
• robotsuite Drop pkg_resources usage collective/robotsuite#30
• zope.publisher Drop pkg resources usage zopefoundation/zope.publisher#84
• zope.testrunner Drop pkg_resources usage zopefoundation/zope.testrunner#193

For the entry points problem, maybe some of it will be fixed when all of the above have been handled. But it may need some fixes in other places. Some issues/PRs:

• TypeError in zc.buildout during installation buildout.coredev#990
• Test with latest setuptools 75.8.2, and with pip 25.0.1. buildout/buildout#677, released in zc.buildout 4.1
• Pkgresources fix + python version update zestsoftware/zest.releaser#456, plus a few follow-up PRs, latest release is 9.4.0.
• Version 4.1 needs current setuptools buildout/buildout#679, fixed by Fix error adding minimum zc.buildout version as requirement. buildout/buildout#680, released in zc.buildout 4.1.1.
• AttributeError: 'NoneType' object has no attribute 'location' buildout/buildout#681, fixed by Fix error finding zc.buildout dist in upgrade check. buildout/buildout#682, released in zc.buildout 4.1.2.
• Problems loading entry points buildout/buildout#683, being fixed in Patch the find method from pkg_resources.WorkingSet. buildout/buildout#684
• Scripts with recent binary wheels may have unfindable distributions buildout/buildout#686. Possible workaround tracked in Restrict setuptools to at most 75.8.0 for building? meta#267

Risks

If we find the correct replacements, this should have no adverse effects.

Participants

[stevepiercy]

I second and will participate. Old packages that use Sphinx to build their docs might need to implement this fix in their conf.py:

https://github.com/plone/plone.api/pull/566/files

[gforcada]

@mauritsvanrees you can add myself as seconder if you need someone :)

It would be nice to have a flake8 plugin for that, so we do not get regressions.

[jensens]

I support this PLIP.

[yurj]

Are you planning to convert code in place or have some utils functions that wraps up the calls to importlib? For example:

- __version__ = pkg_resources.require("Products.CMFPlone")[0].version
+ __version__ = plone.pgkutils.getversion("Products.CMFPlone")
+ from importlib.metadata import version
+ def getversion(packagename):
+
+    return version(packagename)

This thin wrapper can help in the future? Maybe there will be other changes in the packaging world in the future. Moreover, could be this wrap work an help to the future work on the namespace problem?

[mauritsvanrees]

If we have multiple lines that belong together and are used often, then a wrapper function could help. But for 99% of the cases I could prefer to simply call importlib and friends directly.

[jensens]

Are you planning to convert code in place or have some utils functions that wraps up the calls to importlib?

I do not think one layer more is needed here. importlib in Python is stable, it in the standard library and not so part of an add-on.

[gforcada]

• all PRs are done
• plenty need to be merged
• some have test failures that seem related to CMFPlone still using pkg_resources (as it has not been released)
• some test failures are harder to investigate (like on plone.autoinclude) 😓