Add Sliding Sync /sync endpoint (initial implementation)

[MadLittleMods]

Add initial implementation of the Sliding Sync /sync endpoint (not fully functional).

Based on MSC3575: Sliding Sync

This iteration only focuses on returning the list of room IDs in the sliding window API (without sorting/filtering).

Rooms appear in the Sliding sync response based on:

• invite, join, knock, ban membership events
• Kicks (leave membership events where sender is different from the user_id/state_key)
• newly_left (rooms that were left during the given token range, > from_token and <= to_token)
• In order for bans/kicks to not show up, you need to /forget those rooms. This doesn't modify the event itself though and only adds the forgotten flag to room_memberships in Synapse. There isn't a way to tell when a room was forgotten at the moment so we can't factor it into the from/to range.

Example request

POST http://localhost:8008/_matrix/client/unstable/org.matrix.msc3575/sync

{
  "lists": {
    "foo-list": {
      "ranges": [ [0, 99] ],
      "sort": [ "by_notification_level", "by_recency", "by_name" ],
      "required_state": [
        ["m.room.join_rules", ""],
        ["m.room.history_visibility", ""],
        ["m.space.child", "*"]
      ],
      "timeline_limit": 100
    }
  }
}

Response:

{
  "next_pos": "s58_224_0_13_10_1_1_16_0_1",
  "lists": {
    "foo-list": {
      "count": 1,
      "ops": [
        {
          "op": "SYNC",
          "range": [0, 99],
          "room_ids": [
            "!MmgikIyFzsuvtnbvVG:my.synapse.linux.server"
          ]
        }
      ]
    }
  },
  "rooms": {},
  "extensions": {}
}

Dev notes

• Pydantic validation in Synapse: https://github.com/matrix-org/synapse/milestone/9
• https://docs.pydantic.dev/latest/api/types/
• https://docs.pydantic.dev/2.0/usage/types/list_types/

---

Avoid shadowing arguments - disallow reassignment - masking existing variables:

• Fix backfilled events being rejected for no state_groups matrix-org/synapse#10439 (comment)
• Disallow reassignment of function parameters wemake-services/wemake-python-styleguide#2128
• Allow Final in type arguments to avoid shadowing arguments - disallow reassignment of function parameters python/mypy#11076

Pull Request Checklist

• Pull request is based on the develop branch
• Pull request includes a changelog file. The entry should:
  • Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from EventStore to EventWorkerStore.".
  • Use markdown where necessary, mostly for code blocks.
  • End with either a period (.) or an exclamation mark (!).
  • Start with a capital letter.
  • Feel free to credit yourself, by adding a sentence "Contributed by @github_username." or "Contributed by [Your Name]." to the end of the entry.
• Code style is correct
  (run the linters)

[MadLittleMods]

To address failing CI:

poetry run trial tests.rest.client.test_rooms.RoomMemberStateTestCase.test_invalid_puts tests.rest.client.test_rooms

poetry run trial tests.rest.client.test_rooms.RoomMemberStateTestCase.test_invalid_puts
tests.rest.client.test_rooms
  RoomMemberStateTestCase
    test_invalid_puts ...                                                [FAIL]

===============================================================================
[FAIL]
Traceback (most recent call last):
  File "/home/eric/Documents/github/element/synapse/tests/rest/client/test_rooms.py", line 1069, in test_invalid_puts
    self.assertEqual(
  File "/home/eric/.cache/pypoetry/virtualenvs/matrix-synapse-xCtC9ulO-py3.12/lib/python3.12/site-packages/twisted/trial/_synctest.py", line 444, in assertEqual
    super().assertEqual(first, second, msg)
  File "/usr/lib/python3.12/unittest/case.py", line 885, in assertEqual
    assertion_func(first, second, msg=msg)
  File "/usr/lib/python3.12/unittest/case.py", line 878, in _baseAssertEqual
    raise self.failureException(msg)
twisted.trial.unittest.FailTest: <HTTPStatus.BAD_REQUEST: 400> != 500 : b'{"errcode":"M_UNKNOWN","error":"Internal server error"}'

tests.rest.client.test_rooms.RoomMemberStateTestCase.test_invalid_puts
-------------------------------------------------------------------------------
Ran 1 tests in 1.203s

_trial_temp/test.log

2024-06-05 11:49:09-0500 [-] synapse.http.server - 146 - ERROR - PUT-7 - Failed handle request via 'RoomStateEventRestServlet': <SynapseRequest at 0x7f408b359a00 method='PUT' uri='/_matrix/client/r0/rooms/!iqYybTUtBNrlPoqnZY:red/state/m.room.member/@sid1:red' clientproto='1.1' site='red'>
	Traceback (most recent call last):
	  File "/home/eric/Documents/github/element/synapse/synapse/http/server.py", line 332, in _async_render_wrapper
	    callback_return = await self._async_render(request)
	                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	  File "/home/eric/Documents/github/element/synapse/synapse/http/server.py", line 544, in _async_render
	    callback_return = await raw_callback_return
	                      ^^^^^^^^^^^^^^^^^^^^^^^^^
	  File "/home/eric/Documents/github/element/synapse/synapse/rest/client/room.py", line 295, in on_PUT
	    event_id, _ = await self.room_member_handler.update_membership(
	                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	  File "/home/eric/Documents/github/element/synapse/synapse/handlers/room_member.py", line 666, in update_membership
	    result = await self.update_membership_locked(
	             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	  File "/home/eric/Documents/github/element/synapse/synapse/handlers/room_member.py", line 832, in update_membership_locked
	    if effective_membership_state not in Membership.LIST:
	       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	TypeError: unhashable type: 'list'

Previously, we would just pass whatever we received from the event content into the handler layer that expected a plain string.

Since I changed Membership.LIST to a set, when people sent things like arrays/lists, we got unhashable errors. Whereas previously, the comparison worked fine with a Tuple.

This just adds some validation to the rest layer so the handler can always expect a plain str as the type signature says.

[MadLittleMods]

Thanks for the reviews @erikjohnston 🐙

[jsalort]

When running synapse in docker along with a syncv3 service, does this feature possibly clash with the syncv3 service using ghcr.io/matrix-org/sliding-sync ?

[MadLittleMods]

When running synapse in docker along with a syncv3 service, does this feature possibly clash with the syncv3 service using ghcr.io/matrix-org/sliding-sync ?

-- @jsalort, #17187 (comment)

Since this endpoint is currently using /_matrix/client/unstable/org.matrix.simplified_msc3575/sync, I don't think it would clash with the Sliding Sync proxy.