room hierarchy is empty on newly joined spaces, logs spammed with "Unable to get hierarchy of [room] via federation"

[itzzengay]

Description

whenever I join or create a new space in matrix, the room list is seemingly empty despite there being rooms inside. What is shown in element is an example of this bug.

Steps to reproduce

• join or create a new matrix space
• (if you create a space, create a room in that matrix space)
• go to the space's room list, there are no rooms and a blank screen is shown

Homeserver

matrix.constellatory.net

Synapse Version

1.111.0

Installation Method

pip (from PyPI)

Database

PostgreSQL (single postgresql server, no, no)

Workers

Multiple workers

Platform

• Debian GNU/Linux 12 (bookworm) x86_64
• amd64

Configuration

• Presence is enabled
• the mjolnir synapse module is enabled

Relevant log output

Jul 22 03:40:03 gemini matrix-synapse[56411]: 2024-07-22 03:40:03,559 - synapse.handlers.room_summary - 506 - WARNING - GET-27879 - Unable to get hierarchy of !ikZyORXMMTpqykSeJZ:constellatory.net via federation: 502: Failed to fetch room hierarchy via any server
Jul 22 03:40:03 gemini matrix-synapse[56411]: 2024-07-22 03:40:03,559 - synapse.handlers.room_summary - 506 - WARNING - GET-27879 - Unable to get hierarchy of !UCztbsCOduEsvKHvht:constellatory.net via federation: 502: Failed to fetch room hierarchy via any server
Jul 22 03:40:03 gemini matrix-synapse[56411]: 2024-07-22 03:40:03,561 - synapse.handlers.room_summary - 506 - WARNING - GET-27879 - Unable to get hierarchy of !OgRTnvREmECzWFhXLb:constellatory.net via federation: 502: Failed to fetch room hierarchy via any server

Jul 22 03:38:59 gemini matrix-synapse[56411]: 2024-07-22 03:38:59,032 - synapse.http.matrixfederationclient - 856 - WARNING - GET-27760 - {GET-O-386} [grapheneos.org] Request failed: GET matrix-federation://grapheneos.org/_matrix/federation/v1/hierarchy/%21dazAYvfqBmVPUWwGeY%3Agrapheneos.org?suggested_only=false: HttpResponseException('404: Not Found')
Jul 22 03:38:59 gemini matrix-synapse[56411]: 2024-07-22 03:38:59,032 - synapse.handlers.room_summary - 506 - WARNING - GET-27760 - Unable to get hierarchy of !dazAYvfqBmVPUWwGeY:grapheneos.org via federation: 404: Unknown room: !dazAYvfqBmVPUWwGeY:grapheneos.org
Jul 22 03:38:59 gemini matrix-synapse[56411]: 2024-07-22 03:38:59,559 - synapse.http.matrixfederationclient - 856 - WARNING - GET-27760 - {GET-O-387} [grapheneos.org] Request failed: GET matrix-federation://grapheneos.org/_matrix/federation/v1/hierarchy/%21qvbxJeWbwVzENAnSHc%3Agrapheneos.org?suggested_only=false: HttpResponseException('404: Not Found')
Jul 22 03:38:59 gemini matrix-synapse[56411]: 2024-07-22 03:38:59,559 - synapse.handlers.room_summary - 506 - WARNING - GET-27760 - Unable to get hierarchy of !qvbxJeWbwVzENAnSHc:grapheneos.org via federation: 404: Unknown room: !qvbxJeWbwVzENAnSHc:grapheneos.org
Jul 22 03:38:59 gemini matrix-synapse[56411]: 2024-07-22 03:38:59,581 - synapse.http.matrixfederationclient - 856 - WARNING - GET-27760 - {GET-O-388} [grapheneos.org] Request failed: GET matrix-federation://grapheneos.org/_matrix/federation/v1/hierarchy/%21nZuRsqCRvxwcWfzuSg%3Agrapheneos.org?suggested_only=false: HttpResponseException('404: Not Found')
Jul 22 03:38:59 gemini matrix-synapse[56411]: 2024-07-22 03:38:59,581 - synapse.handlers.room_summary - 506 - WARNING - GET-27760 - Unable to get hierarchy of !nZuRsqCRvxwcWfzuSg:grapheneos.org via federation: 404: Unknown room: !nZuRsqCRvxwcWfzuSg:grapheneos.org

Jul 22 04:01:01 gemini matrix-synapse[56411]: 2024-07-22 04:01:01,855 - synapse.handlers.room_summary - 506 - WARNING - GET-30097 - Unable to get hierarchy of !8whYSc42cBzkWDHo:solarpunk.moe via federation: 404: room is unknown/forbidden
Jul 22 04:01:02 gemini matrix-synapse[56411]: 2024-07-22 04:01:02,110 - synapse.http.matrixfederationclient - 856 - WARNING - GET-30097 - {GET-O-458} [solarpunk.moe] Request failed: GET matrix-federation://solarpunk.moe/_matrix/federation/v1/hierarchy/%2161PmubtUjJmpeuAg%3Asolarpunk.moe?suggested_only=false: HttpResponseException('404: Not Found')

Anything else that would be useful to know?

n/a

[devonh]

I'm unable to reproduce this using a monolith setup which makes me believe there is an issue with your worker setup or ingress handling.
Would you be able to provide more information on what your worker configuration is?

Also, did you have this working before v1.111.0?

[itzzengay]

I will be able to submit my entire synapse config + the worker configurations in a bit, I'll send it as a zip once I'm at my desktop. And, yes, I had this working before synapse version v1.111.0. I will mention that rolling back to a previous version doesn't fix the issue, and I haven't made any configuration changes in between this version and now. --- "You and I are gonna live forever" Jul 23, 2024, 20:12 by ***@***.***:

…

I'm unable to reproduce this using a monolith setup which makes me believe there is an issue with your worker setup or ingress handling. Would you be able to provide more information on what your worker configuration is? Also, did you have this working before v1.111.0? — Reply to this email directly, > view it on GitHub <#17463 (comment)>> , or > unsubscribe <https://github.com/notifications/unsubscribe-auth/ATM4XCNVWJF5DNPWZXRTRQDZN3WPXAVCNFSM6AAAAABLHPYZL6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENBWGYYTCMJUGY>> . You are receiving this because you authored the thread.> Message ID: > <element-hq/synapse/issues/17463/2246611146> @> github> .> com>

[itzzengay]

it appears this was all caused by a misconfiguration in my background worker cuz I typo'd the worker name X/, fixed that and everything is starting to work again... oops