Install Python 3.10 on all nodes

[mauritsvanrees]

Zope is almost compatible with Python 3.10. See comment, although I don't know if all individual packages are tested yet. But this means we could start testing how well Plone does on 3.10.

There are at least two things that I can think of that first need to happen:

• Make Python 3.10 available on the nodes.
• Create a normal job and a PR job for Plone 6.0 with Python 3.10. For now, they should only be started manually. We don't want to test 3.10 for all PRs and commits yet.

Anything else?

I have no idea how much work it is to fix Plone for 3.10. I have not tried anything yet.
If it is easy, it would be interesting to officially support 3.10 when 6.0 final comes out. But currently it would be fine to support 3.9 as highest Python. So no pressure.

[gforcada]

I will try to work on that during the weekend, it should mostly take time as it compiles python to get it to run faster.

[gforcada]

I was worse than I expected during the weekend, I got my second vaccination, and now that I finally I'm human again, I'm already starting the process: python 3.10 is already added on https://github.com/gforcada/ansible-compile-python and jenkins nodes are starting to compile python :)

[gforcada]

I'm rather surprised that it went so fast, but 3.10 is already on the nodes 🎉 I will create a jenkins job and set it to run 🤞🏾

[gforcada]

😱 it fails not on any plone code but rather at trying to create a virtual environment for it to run on 😕 the blame should go on shiningpanda jenkins plugin... we used this plugin specially because it was handling the nice thing to have a separate virtual environment for every job running, so we could install dependencies etc etc without us having to manually maintain the virtual environment...

We might have to switch back to it, at least for 3.10 onwards 😕

[gforcada]

I created an upstream ticket: https://issues.jenkins.io/browse/JENKINS-67305

[mauritsvanrees]

Thanks for your effort so far, Gil!

[fredvd]

@gforcada When I searched on that last error line I found the following issue on pip's repo: pypa/pip#10253

Apparently that traceback was solved in pip 21.2.3. . Which version of pip is being used to create the virtualenv in the plugin is that configurable? Maybe we need to install that updated pip first in the global python 3.10, I'm not sure where virtualenv or 'python3.10 -mvenv` get their pip from that they install into the venv.

[gforcada]

@fredvd thanks for looking that up, I noticed though that this is coming from the bootstrap script from the shiningpanda plugin itself: https://github.com/jenkinsci/shiningpanda-plugin/blob/f4b5d326b6cec5ef09edc54e303b2b30aa79277b/src/main/resources/jenkins/plugins/shiningpanda/workspace/virtualenv.py#L1521

I see that it always downloads and install pip/setuptools/wheel with the following versions:
wheel: '0.31.1'
setuptools: '39.1.0'
pip: '10.0.1'

[fredvd]

Ouch, those are terribly old versions. I was looking at the python file to see any version numbers, but the 3 wheels are in the repo directory as raw archives with a 'zero' version. Off course in 2018 Python 2 was still much more in use.

I'm amazed that previous Python versions worked at all... But then we don't really use pip extensively when building buildout 2. (allthough I found out this month resolving sub-dependencies of zope.packages does insert wheels in my local pip cache, even with zc.buildout 2... different story)

I don't think shiningpandas is still very active as a company / saas service. Their last blog / updated item on the website is from 2018 :-O

There are quite a bit of forks on the jenkins plugin repo, but these also seem to be 3> years old. So we might have to squeeze and bend the plugin ourselves to get it working with 3.10

[gforcada]

If you look at the usage: https://plugins.jenkins.io/ui/search?query=python the python one, either is strong solid as it can get, because it did not had a release for the last 6 years, but at the same time it is used more than 3 times the shiningpanda one.

So I guess we will have to turn all our shiningpanda wrapped scripts into plain scripts and manage the virtualenvs ourselves 🤷🏾

At least, if we only do it for the python3 jobs, it is not that cumbersome to do anyway, but still more code to write and maintain 😑

[mauritsvanrees]

Latest run on Jenkins is green. Yeehah!

Is there anything left to do here? Python 3.10 seems to be installed everywhere, at least on the active nodes.

There is some followup that we could do, but that can be a new issue:

• Create a PR job for 3.10.
• Split the current 3.10 job into Python tests and Robot tests. (The current one only runs bin/test, so no robot tests.)
• Setup mr.roboto to start these jobs.

[gforcada]

Sounds like a plan, I don't have much time, but I will want to get out of my tasks every now and then, so I will try my best to set up all this, not sure when though, I'm fine reviewing changes from others though 👍🏾 😃

[fredvd]

@gforcada . @mauritsvanrees and me just tuned some settings on robotframework tests and percentage of failed tests to be counted as unstable and we added Python3.10 Plone 6 robotframework tests.

We get the same error as we had in december though as reported in this issue with shiningpandas and the pip 21.x issue .

But if I remember correctly you did change something later in the scripts to fix this, or? The Normal Python3.10 Plone 6.0 was running fine, until I updates the jobs from this repo. Was something changed maybe manually on jenkins in between?

[edit] Oh, apparently now only the Python 3.10 jobs are not working correctly. I first thought the Python 3.9 jobs for Plone 6.0 were also broken after I pushed the jobs from this repo, but that is not the case.

I'll split the Python 3.10 / Plone 6 jobs into a new job template and try to set that up using the normal Python jenkins plugins and script and not shiningpandas, as discussed above.

[gforcada]

Sorry to come back to this one this late 😅

I, manually, replaced the scripts on the 2 jenkins jobs for Plone 6.0 on python 3.10 and they are running as I type:

https://jenkins.plone.org/job/plone-6.0-python-3.10-robot-chrome/315/
https://jenkins.plone.org/job/plone-6.0-python-3.10/672/

So far they are installing dependencies, which is much more than what it used to do until then 😃

[petschki]

Seems that the jobs are green now. Should we enable py3.10 for PRs also? or too early...

[mauritsvanrees]

Great to have 3.10 back. Thanks!

Is this safe? Meaning, if you or Fred automatically updates the server with new versions of the scripts, are your changes then kept?

I have a PR which passed on 3.7, 3.8. 3.9. I have manually started 3.10 for it, but that fails within a few seconds:

Using base prefix '/srv/python3.10'
New python executable in /home/jenkins/shiningpanda/jobs/88954158/virtualenvs/0dd3bfc1/bin/python3.10
Also creating executable in /home/jenkins/shiningpanda/jobs/88954158/virtualenvs/0dd3bfc1/bin/python
Traceback (most recent call last):
  File "/home/jenkins/shiningpanda/jobs/88954158/virtualenv.py", line 2343, in <module>
    main()
  File "/home/jenkins/shiningpanda/jobs/88954158/virtualenv.py", line 703, in main
    create_environment(home_dir,
  File "/home/jenkins/shiningpanda/jobs/88954158/virtualenv.py", line 925, in create_environment
    py_executable = os.path.abspath(install_python(
  File "/home/jenkins/shiningpanda/jobs/88954158/virtualenv.py", line 1438, in install_python
    fix_local_scheme(home_dir, symlink)
  File "/home/jenkins/shiningpanda/jobs/88954158/virtualenv.py", line 1521, in fix_local_scheme
    if sysconfig._get_default_scheme() == 'posix_local':
AttributeError: module 'sysconfig' has no attribute '_get_default_scheme'. Did you mean: 'get_default_scheme'?

[gforcada]

@mauritsvanrees that's the PR job, I did not fix that one, I guess I should 😄

as for the configuration changes, no, it is not on the jenkins.plone.org repo, they will be lost.

We could simplify all python 3 scripts by avoiding using the shining panda plugin. It was nice as long as it worked, but provides not so much extra benefits, basically it automatically creates a venv and gives you a script were it is already enabled.

In python 3 that's only 2 lines 😄

[gforcada]

@mauritsvanrees the PR job is running: https://jenkins.plone.org/job/pull-request-6.0-3.10/4/console

btw. I did some cleanups on the extra CSS we were adding on top of jenkins own CSS, it should be all fine already :)

[mauritsvanrees]

Thanks!
Getting rid of Shining Panda would be good. Well, I can't really oversee the consequences. And maybe it is needed for Python 2.7.

[gforcada]

For now, if we don't want to add 3.11 on the mix, I guess not, I would leave it until we release 6.0 and a month or so, for last minute fixes.

Then we can do the changes, or would it be ok to do the changes nowish? I will be on vacations the next 2 weeks, so I probably have some time (maybe 😓 )

[mauritsvanrees]

For me it would be good to do changes now. Otherwise I fear your changes for 3.10 will get lost again.

[gforcada]

I will try to do it on the next days/weeks then, though it will be quite a bit of a refactoring 👍🏾

[gforcada]

I pushed a new release of https://github.com/gforcada/ansible-compile-python which adds newer versions of 3.7 to 3.10 and I also added 3.11.0rc1, I created, manually, a jenkins job for it which is expected to fail, as it did 😅

We need some zope releases out first. I guess until a 3.11 final release is out they will not make them, so I'm not sure how useful is it to keep this job, as tracking master branches for an unreleased version might provide more work than benefit 🤷🏾

[mauritsvanrees]

Zope is being updated to 3.11, with new version pins. I asked about the status today. One idea that I express there, is that we could add [sources:python311] and override the Zope checkout to use the branch of that PR. I did not try it yet, and don't know if buildout supports this yet for 3.11.

[gforcada]

Meanwhile 3.10 and 3.11alpha are already installed. I'm closing this one 😄

[mauritsvanrees]

Thanks for your work!

[wesleybl]

@mauritsvanrees @gforcada

Is something missing to enable testing in Python 3.10 in PRs? Any predictions for that?

[gforcada]

@wesleybl for Plone 6.0 there is already a job for that what else are you missing? 🤔

[wesleybl]

@gforcada when we open a PR, only Jenkins test from Python 3.7 to 3.9 appears in Some checks haven't completed yet. For example:

plone/Products.CMFPlone#3540 (comment)

[stevepiercy]

See also plone/plone.restapi#1485.

[mauritsvanrees]

I see for example this error:

[pull-request-6.0-3.10] $ /bin/sh /tmp/jenkins393430905353911877.sh
/tmp/jenkins393430905353911877.sh: 4: Bad substitution

@gforcade Could that be this line, that /srv/python${{python_version}} needs to have less braces, /srv/python${python_version}?
In 5.x-tests.sh it looks the same, but that is included raw, so maybe that makes it okay.

I see the normal 5.2-3.6 job failing too, which seems a very different error though:

[plone-5.2-python-3.6] $ /bin/sh /tmp/jenkins5781342866443840908.sh
Error: Command '['/home/jenkins/workspace/plone-5.2-python-3.6/venv/bin/python3', '-Im', 'ensurepip', '--upgrade', '--default-pip']' died with <Signals.SIGSEGV: 11>.
/tmp/jenkins5781342866443840908.sh: 5: .: cannot open venv/bin/activate: No such file
Build step 'Execute shell' marked build as failure

[gforcada]

😱 oh my, it was working a few days ago after I did the changes... 🤔

[gforcada]

@fredvd oh, I see, on PR tests indeed there are too much brackets 😕

[gforcada]

@mauritsvanrees that was meant for you 😅 PR jobs seem to be fixed, but I noticed that python 3.6 can not create virtual environments on its own, the venv module was added before, or? 🤔

[gforcada]

Seems that it happens only on 3.6...

[mauritsvanrees]

venv should be available in 3.6. On my Mac:

$ python3.6 -mvenv foo
$ cd foo
$ . bin/activate

Ah, maybe with the newer Ubuntu version the 3.6 can no longer find the venv module? Maybe a package with a venv specifically for 3.6 is available? Just guessing.

[gforcada]

We are compiling the python versions, for speed reasons, but somehow it crashes (a missing dependency I guess)

[gforcada]

indeed, also on robot tests, the 3.6 can not start... I will look at it later if that's fine

[mauritsvanrees]

PR jobs still fail, for example 6.0-310:

[pull-request-6.0-3.10] $ /bin/sh /tmp/jenkins5295237807733282660.sh
/tmp/jenkins5295237807733282660.sh: 4: /srv/python{py}/bin/python3: not found
/tmp/jenkins5295237807733282660.sh: 5: .: cannot open venv/bin/activate: No such file

[gforcada]

Oh, I see 😕 sorry for the breakage 😖

[gforcada]

I fixed the PR jobs 🎉

[gforcada]

robot tests jobs and python3.6 seem to be failing still 🤔

[wesleybl]

It seems that on node4 the ChromeDriver version has to be updated. All Robots tests fail with the error:

Setup failed:
SessionNotCreatedException: Message: session not created: This version of ChromeDriver only supports Chrome version 100
Current browser version is 105.0.5195.102 with binary path /usr/bin/google-chrome

See: https://jenkins.plone.org/job/pull-request-6.0-3.8/1884/consoleFull

node4 appears to be an Ubuntu18.04

Already node1 is an Ubuntu 22.04 and on it only 12 Robots tests fail. The browser seems to work on that node.

See: https://jenkins.plone.org/job/pull-request-6.0-3.10/24/console

Could it be Chrome version related glitches?

[gforcada]

@wesleybl all noes are on ubuntu 22.04 already, I upgraded them last week of August, but there is indeed some glitches with running robot tests :-/

[wesleybl]

@wesleybl all noes are on ubuntu 22.04 already, I upgraded them last week of August, but there is indeed some glitches with running robot tests :-/

A job run on 2022-09-09 on node4, showed that it was ubuntu 18.04:

Building remotely on [Node4](https://jenkins.plone.org/computer/Node4) (syslab Ubuntu18.04) in workspace /home/jenkins/workspace/pull-request-6.0-3.8

See: https://jenkins.plone.org/job/pull-request-6.0-3.8/1884/consoleFull

But latest node 4 jobs already show Ubuntu 22: https://jenkins.plone.org/job/pull-request-6.0-3.8/1894/consoleFull

However, the error:

SessionNotCreatedException: Message: session not created: This version of ChromeDriver only supports Chrome version 100
Current browser version is 105.0.5195.102 with binary path /usr/bin/google-chrome

still continues on node 4. ChromeDriver version needs to be updated.

I also see the error:

Traceback (most recent call last):
  File "/home/jenkins/workspace/pull-request-6.0-3.10/templates/pr-update-status-py3.py", line 5, in <module>
    from github import Github
ModuleNotFoundError: No module named 'github'

in all jobs. Looks like I'm missing some package in Python. Jenkins is not notifying github about hits or misses.

[gforcada]

this syslab ubuntu18.04 are tags that one set manually on jenkins, I already updated them to show 22.04 😄

thanks for having a look and reporting back! ✨ the github error I noticed only yesterday myself

as for the chromedriver I did download and install the very same, of course something is still off 😖

[wesleybl]

as for the chromedriver I did download and install the very same, of course something is still off confounded

Is the chrome version the same on all nodes?

One thing that would be nice is the job to print the version of chrome and chromedriver:

chromedriver --version
google-chrome --version

[davisagli]

@gforcada How did you install chromedriver? Maybe there is another copy of it on node4 that's being used instead? I am willing to investigate it, but I don't know how to access the node.

[gforcada]

indeed the driver and google-chrome versions differ, on node1 for almost nothing, the other two (node3 and node4) are a bit more behind, but that's because I was testing only first on node1.

@davisagli I installed your public key (as found in github) on node1 server. I'm sending an email with the details, thanks a lot, another pair of hands will be more than welcome 😄 I've been really busy these last weeks, and doing it at midnight does not help focusing 😓

[gforcada]

I did have some time and, hopefully on node1 all robot jobs should work ™️ 🤞🏾

[davisagli]

I think we need #307 first

[mauritsvanrees]

307 was merged and released. It now mostly works for a 6.0 PR. Thanks!
The only thing going wrong there is that Python 3.10 on Node 1 is still missing pygithub. See my comment.
Does this need another fix in a script?

[davisagli]

I fixed the status update for the Python 3.10 build by updating pygithub to a newer version.

[mauritsvanrees]

Seems good, thank you both!

Two open points that we can track outside this bug:

• PR Removed Python 3.7 from Plone 6.0 jobs. #309 removed 3.7 from the 6.0 jobs in the config
• issue Plone 5.2 Python 3.6 jobs fail #310 is that Plone 5.2 Python 3.6 jobs fail to start