Infinite loop on startup due to jsonschema 4.5.0

[progval]

Description

Steps to reproduce

• start with a stock Debian 11
• pip3 install matrix-synapse (also work on current develop branch)
• start synapse (python3 -m synapse.app.homeserver --config-path ...)

This causes Synapse to enter an infinite loop before binding any port. Terminating the process produces this traceback:

Traceback (most recent call last):
  File "/usr/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/dev-matrix/synapse/synapse/app/homeserver.py", line 460, in <module>
    main()
  File "/home/dev-matrix/synapse/synapse/app/homeserver.py", line 450, in main
    hs = setup(sys.argv[1:])
  File "/home/dev-matrix/synapse/synapse/app/homeserver.py", line 326, in setup
    config = HomeServerConfig.load_or_generate_config(
  File "/home/dev-matrix/synapse/synapse/config/_base.py", line 697, in load_or_generate_config
    obj.parse_config_dict(
  File "/home/dev-matrix/synapse/synapse/config/_base.py", line 718, in parse_config_dict
    self.invoke_all(
  File "/home/dev-matrix/synapse/synapse/config/_base.py", line 347, in invoke_all
    res[config_class.section] = getattr(config, func_name)(*args, **kwargs)
  File "/home/dev-matrix/synapse/synapse/config/oembed.py", line 47, in read_config
    self.oembed_patterns: List[OEmbedEndpointConfig] = list(
  File "/home/dev-matrix/synapse/synapse/config/oembed.py", line 63, in _parse_and_validate_providers
    yield from self._parse_and_validate_provider(
  File "/home/dev-matrix/synapse/synapse/config/oembed.py", line 86, in _parse_and_validate_provider
    validate_config(
  File "/home/dev-matrix/synapse/synapse/config/_util.py", line 40, in validate_config
    jsonschema.validate(config, json_schema)
  File "/home/dev-matrix/.local/lib/python3.9/site-packages/jsonschema/validators.py", line 1036, in validate
    cls.check_schema(schema)
[many more frames in jsonschema]
KeyboardInterrupt

I managed to reproduce the issue with this code:

import json, jsonschema
from synapse.config.oembed import _OEMBED_PROVIDER_SCHEMA
config = json.load(open('../synapse/synapse/res/providers.json'))
jsonschema.validate(config, _OEMBED_PROVIDER_SCHEMA)

which reproduces the infinite loop

Version information

• Homeserver: local

• Version: 1.58.1 or current develop

• Install method: pip3 install matrix-synapse or pip3 install -e .

• Platform: Debian 11 (Python 3.9)

$ pip3 freeze
arandr==0.1.10
attrs==21.4.0
Automat==20.2.0
bcrypt==3.2.2
beautifulsoup4==4.9.3
black==22.3.0
bleach==5.0.0
borgbackup==1.1.16
Brlapi==0.8.2
canonicaljson==1.6.1
certifi==2020.6.20
cffi==1.15.0
chardet==4.0.0
chrome-gnome-shell==0.0.0
click==8.1.3
colorama==0.4.4
constantly==15.1.0
cryptography==3.3.2
cupshelpers==1.0
dbus-python==1.2.16
decorator==4.4.2
distro==1.5.0
distro-info==1.0
frozendict==2.3.2
gpg===1.14.0-unknown
gyp==0.1
html5lib==1.1
httplib2==0.18.1
hyperlink==21.0.0
idna==2.10
ijson==3.1.4
incremental==21.3.0
iniconfig==1.1.1
iotop==0.6
Jinja2==3.1.2
jsonschema==4.5.0
llfuse==1.3.8
louis==3.16.0
lxml==4.6.3
Mako==1.1.3
MarkupSafe==1.1.1
matrix-common==1.1.0
matrix-synapse==1.58.1
msgpack==1.0.3
mypy-extensions==0.4.3
netaddr==0.8.0
olefile==0.46
packaging==20.9
pathspec==0.9.0
pexpect==4.8.0
phonenumbers==8.12.47
Pillow==8.1.2
platformdirs==2.5.2
pluggy==1.0.0
powerline-status==2.8.1
prometheus-client==0.14.1
psutil==5.8.0
py==1.11.0
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycairo==1.16.2
pycparser==2.21
pycups==2.0.1
pycurl==7.43.0.6
PyGObject==3.38.0
pyinotify==0.9.6
pymacaroons==0.13.0
PyNaCl==1.5.0
pyOpenSSL==22.0.0
pyparsing==2.4.7
PyQt5==5.15.2
PyQt5-sip==12.8.1
pyrsistent==0.18.1
PySimpleSOAP==1.16.2
pysmbc==1.0.23
PySocks==1.7.1
pyte==0.8.0
pytest==7.1.2
python-apt==2.2.1
python-debian==0.1.39
python-debianbts==3.1.0
python-pam==1.8.4
python-xapp==2.0.2
python-xlib==0.29
pytz==2021.1
pyxdg==0.27
PyYAML==6.0
reportbug==7.10.3+deb11u1
requests==2.25.1
service-identity==21.1.0
setproctitle==1.2.1
signedjson==1.1.4
simplejson==3.17.6
six==1.16.0
sortedcontainers==2.4.0
soupsieve==2.2.1
SQLAlchemy==1.3.22
tinycss==0.4
tinycss2==1.0.2
tomli==2.0.1
torbrowser-launcher==0.3.3
treq==22.2.0
Twisted==22.4.0
typing_extensions==4.2.0
unattended-upgrades==0.1
unpaddedbase64==2.1.0
urllib3==1.26.5
wcwidth==0.1.9
webencodings==0.5.1
wxPython==4.0.7
xdg==5
zope.interface==5.4.0

[DMRobertson]

Interesting. Wasn't reproducible (from the snippet) for me locally (jsonschema 4.4.0, on a377a43.

[DMRobertson]

...but is reproducible on 4.5.0 (which is two hours old as I write this).

https://github.com/python-jsonschema/jsonschema/releases/tag/v4.5.0 doesn't have anything that looks obviously suspicious though.

[clokep]

Maybe python-jsonschema/jsonschema#941?

[progval]

indeed, downgrading to 4.4.0 fixes the loop for me.

Should I report it to jsonschema, then?

[DMRobertson]

I think Patrick's found the right issue in jsonschema. I'll see if I can minimise the example you've found and add it to that issue.

(I hope the latestdeps job finds this tomorrow morning!)

We could temporarily pin to version <=4.5.0 in the meantime I suppose?

[gramakri]

We hit the same issue in the Cloudron package. On startup, the server just hangs. Using synapse 1.58.1 .

[anoadragon453]

jsonschema v4.5.1 has been released a few hours ago, which looks to mitigate this issue, at least in local testing.

To upgrade to it, run pip install -U jsonschema==4.5.1 in your Synapse virtual environment.

We can still exclude jsonschema==4.5.0 from our dependencies if we think it's necessary.