Fix flaky session and thread_runner tests

[merelcht]

Description

There's a couple of flaky test that cause builds to fail. Usually the builds are fine again on re-run.

Development notes

1. Test test_on_node_error_hook_parallel_runner in tests/framework/session/test_session_extension_hooks.py

Kept failing with:

Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.10.16/x64/lib/python3.10/multiprocessing/queues.py", line 244, in _feed
    obj = _ForkingPickler.dumps(obj)
  File "/opt/hostedtoolcache/Python/3.10.16/x64/lib/python3.10/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
TypeError: 'NoneType' object is not callable

Initially I tried to update the logging object to make it pickleable, but this caused other issues. Now I changed the number of workers which seems to do the trick. My guess is that the issue was indeed in logs not being pickleable and being sent across different processes.

2. Test test_stricter_suggest_resume_scenario in tests/runner/test_thread_runner.py

Issue with the size of test_pipelines changing during the test, causing the expected error message "test exception" to not match with the exception caused by the iterable changing size, which is "dictionary changed size during iteration". I have a feeling this also has to do with the fact that there are multiple threads, so set max workers to 1.

Developer Certificate of Origin

We need all contributions to comply with the Developer Certificate of Origin (DCO). All commits must be signed off by including a Signed-off-by line in the commit message. See our wiki for guidance.

If your PR is blocked due to unsigned commits, then you must follow the instructions under "Rebase the branch" on the GitHub Checks page for your PR. This will retroactively add the sign-off to all unsigned commits and allow the DCO check to pass.

Checklist

• Read the contributing guidelines
• Signed off each commit with a Developer Certificate of Origin (DCO)
• Opened this PR as a 'Draft Pull Request' if it is work-in-progress
• Updated the documentation to reflect the code changes
• Added a description of this change in the RELEASE.md file
• Added tests to cover my changes
• Checked if this change will affect Kedro-Viz, and if so, communicated that with the Viz team

[ankatiyar]

Thank you @merelcht! This has been failing almost every day 🙏🏾 🙌🏾

[DimedS]

Thank you, @merelcht !