Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improve DB performance of calculating badge counts for push. #16756

Merged
merged 4 commits into from
Jan 11, 2024
Merged

Improve DB performance of calculating badge counts for push. #16756

merged 4 commits into from
Jan 11, 2024

Conversation

erikjohnston
Copy link
Member

@erikjohnston erikjohnston commented Dec 21, 2023 •
edited
Loading

The crux of the change is to try and make the queries simpler and pull out fewer rows. Before, there were quite a few joins against subqueries, which caused postgres to pull out more rows than necessary.

Instead, let's simplify the query and do some of the filtering out in Python instead, letting Postgres do better optimizations now that it doesn't have to deal with joins against subqueries.

Review note: this is a complete rewrite of the function, so not sure how useful the diff is.

@erikjohnston
Improve DB performance of calculating badge counts for push.
da91285 
The crux of the change is to try and make the queries simpler and pull
out fewer rows. Before, there were quite a few joins against subqueries,
which caused postgres to pull out more rows than necessary.

Instead, let's simplify the query and do some of the filtering out in
Python instead, letting Postgres do better optimizations now that it
doesn't have to deal with joins against subqueries.
@erikjohnston
Newsfile
2c270c9
@erikjohnston erikjohnston marked this pull request as ready for review December 21, 2023 14:58
@erikjohnston erikjohnston requested a review from a team as a code owner December 21, 2023 14:58
anoadragon453
anoadragon453 reviewed Jan 8, 2024
View reviewed changes
Copy link
Member

@anoadragon453 anoadragon453 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mostly looks sane. I'll probably need another pass to double-check the SQL, so for now I leave you with a small question.

synapse/storage/databases/main/event_push_actions.py Outdated Show resolved Hide resolved
synapse/storage/databases/main/event_push_actions.py Outdated Show resolved Hide resolved
synapse/storage/databases/main/event_push_actions.py Show resolved Hide resolved
anoadragon453
anoadragon453 approved these changes Jan 11, 2024
View reviewed changes
Copy link
Member

@anoadragon453 anoadragon453 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Otherwise LGTM!

synapse/storage/databases/main/event_push_actions.py Show resolved Hide resolved
@erikjohnston erikjohnston enabled auto-merge (squash) January 11, 2024 11:37
@erikjohnston erikjohnston merged commit b11f7b5 into develop Jan 11, 2024
37 of 38 checks passed
@erikjohnston erikjohnston deleted the erikj/better_push_badge_perf branch January 11, 2024 11:52
yingziwu added a commit to yingziwu/synapse that referenced this pull request Feb 1, 2024
@yingziwu
Merge tag 'v1.100.0' into develop
1eca755 
No significant changes since 1.100.0rc3.

- Fix database performance regression due to changing Postgres table statistics. Introduced in v1.100.0rc1. ([\#16849](element-hq/synapse#16849))

This version is the same as 1.100.0rc1 but with fixes to the release process.

- Downgrade the `download-artifact` and `upload-artifact` actions to v3 due to breaking changes. ([\#16847](element-hq/synapse#16847))

*This version was never released to PyPI or the Debian repository due to failures in the automatic part of the release process.*

- Advertise experimental support for [MSC4028](matrix-org/matrix-spec-proposals#4028) through `/_matrix/clients/versions` if enabled. Contributed by @hanadi92. ([\#16787](element-hq/synapse#16787))

- Handle wildcard type filters properly for room messages endpoint. Contributed by Mo Balaa. ([\matrix-org#14984](element-hq/synapse#14984))

- Add a link to the "Request log format" explainer on the "Logging sample config" documentation page. ([\#16778](element-hq/synapse#16778))
- Fix broken links in issue templates and documentation. ([\#16810](element-hq/synapse#16810))
- NGINX listen http2 deprecation in documentation template for reverse proxy. ([\#16831](element-hq/synapse#16831))

- Faster partial join to room with complex auth graph. ([\matrix-org#7](element-hq/synapse#7))
- Improve DB performance of calculating badge counts for push. ([\matrix-org#16756](element-hq/synapse#16756))
- Split up deleting devices into batches. ([\matrix-org#16766](element-hq/synapse#16766))
- Remove CI check for sign-off as we require a CLA signature instead. ([\#16776](element-hq/synapse#16776))
- Ensure CI fails when linting fails to make sure auto-merge does the correct thing. ([\#16781](element-hq/synapse#16781))
- Faster load recents for sync by reducing amount of state pulled out. ([\#16783](element-hq/synapse#16783))
- Reduce amount of state pulled out when querying federation hierachy. ([\#16785](element-hq/synapse#16785))
- Pull less state out of the DB when we retry fetching old events during backfill. ([\#16788](element-hq/synapse#16788))
- Optimize query for fetching to-device messages in `/sync`. ([\#16805](element-hq/synapse#16805))
- Reject OIDC config when `client_secret` isn't specified, but the auth method requires one. ([\#16806](element-hq/synapse#16806))
- Allow room creation but not publishing to continue if room publication rules are violated when creating
  a new room. ([\#16811](element-hq/synapse#16811))
- Bump minimum supported Rust version to 1.65.0. ([\#16818](element-hq/synapse#16818))
- Fixup copyright lines in file headers after the licensing change. ([\#16820](element-hq/synapse#16820))
- Add a `--generate-only` option to the internal configuration/launch script for Complement. ([\#16828](element-hq/synapse#16828))
- Preparatory work for tweaking performance of auth chain lookups. ([\#16833](element-hq/synapse#16833))
- Speed up e2e device keys queries for bot accounts. ([\#16841](element-hq/synapse#16841))

* Bump actions/cache from 3 to 4. ([\#16832](element-hq/synapse#16832))
* Bump actions/download-artifact from 3 to 4. ([\#16795](element-hq/synapse#16795))
* Bump actions/upload-artifact from 3 to 4. ([\#16796](element-hq/synapse#16796))
* Bump anyhow from 1.0.75 to 1.0.79. ([\#16789](element-hq/synapse#16789))
* Bump authlib from 1.2.1 to 1.3.0. ([\#16801](element-hq/synapse#16801))
* Bump dawidd6/action-download-artifact from 2.28.0 to 3.0.0. ([\#16794](element-hq/synapse#16794))
* Bump immutabledict from 4.0.0 to 4.1.0. ([\#16812](element-hq/synapse#16812))
* Bump isort from 5.13.1 to 5.13.2. ([\#16835](element-hq/synapse#16835))
* Bump lxml from 4.9.3 to 5.1.0. ([\#16813](element-hq/synapse#16813))
* Bump pillow from 10.1.0 to 10.2.0. ([\#16802](element-hq/synapse#16802))
* Bump pydantic from 2.5.2 to 2.5.3. ([\#16836](element-hq/synapse#16836))
* Bump pyo3 from 0.20.0 to 0.20.2. ([\#16791](element-hq/synapse#16791))
* Bump regex from 1.9.6 to 1.10.3. ([\#16837](element-hq/synapse#16837))
* Bump ruff from 0.1.13 to 0.1.14. ([\#16838](element-hq/synapse#16838))
* Bump ruff from 0.1.7 to 0.1.13. ([\#16814](element-hq/synapse#16814))
* Bump sentry-sdk from 1.35.0 to 1.39.1. ([\#16799](element-hq/synapse#16799))
* Bump serde_json from 1.0.108 to 1.0.111. ([\#16792](element-hq/synapse#16792))
* Bump service-identity from 23.1.0 to 24.1.0. ([\#16816](element-hq/synapse#16816))
* Bump types-commonmark from 0.9.2.4 to 0.9.2.20240106. ([\#16797](element-hq/synapse#16797))
* Bump types-jsonschema from 4.20.0.0 to 4.20.0.20240105. ([\#16800](element-hq/synapse#16800))
* Bump types-jsonschema from 4.20.0.20240105 to 4.21.0.20240118. ([\#16834](element-hq/synapse#16834))
* Bump types-netaddr from 0.9.0.1 to 0.10.0.20240106. ([\#16839](element-hq/synapse#16839))
* Bump typing-extensions from 4.8.0 to 4.9.0. ([\#16815](element-hq/synapse#16815))
@clokep clokep mentioned this pull request Feb 12, 2024
Closed
@ara4n ara4n mentioned this pull request Feb 15, 2024
Closed
anoadragon453 added a commit that referenced this pull request Feb 23, 2024
@anoadragon453
Revert "Improve DB performance of calculating badge counts for push. (#…
ecb03c8 
…16756)"

This reverts commit b11f7b5.
@anoadragon453 anoadragon453 mentioned this pull request Feb 23, 2024
Merged
netbsd-srcmastr pushed a commit to NetBSD/pkgsrc that referenced this pull request Mar 3, 2024
@Midar
Update chat/matrix-synapse to 1.101.0
0b2ea45 
!!! THIS CHANGES THE LICENSE TO AGPLv3 !!!


# Synapse 1.101.0 (2024-02-13)

### Bugfixes

- Fix performance regression when fetching auth chains from the DB. Introduced in v1.100.0. ([\#16893](element-hq/synapse#16893))




# Synapse 1.101.0rc1 (2024-02-06)

### Improved Documentation

- Fix broken links in the documentation. ([\#16853](element-hq/synapse#16853))
- Update MacOS installation instructions to mention that libicu is optional. ([\#16854](element-hq/synapse#16854))
- The version picker now correctly lists versions after `v1.98.0`. ([\#16880](element-hq/synapse#16880))

### Internal Changes

- Add support for stabilised [MSC3981](matrix-org/matrix-spec-proposals#3981) that adds a `recurse` parameter on the `/relations` API. ([\#16842](element-hq/synapse#16842))



### Updates to locked dependencies

* Bump dorny/paths-filter from 2 to 3. ([\#16869](element-hq/synapse#16869))
* Bump gitpython from 3.1.40 to 3.1.41. ([\#16850](element-hq/synapse#16850))
* Bump hiredis from 2.2.3 to 2.3.2. ([\#16862](element-hq/synapse#16862))
* Bump jsonschema from 4.20.0 to 4.21.1. ([\#16887](element-hq/synapse#16887))
* Bump lxml-stubs from 0.4.0 to 0.5.1. ([\#16885](element-hq/synapse#16885))
* Bump mypy-zope from 1.0.1 to 1.0.3. ([\#16865](element-hq/synapse#16865))
* Bump phonenumbers from 8.13.26 to 8.13.29. ([\#16868](element-hq/synapse#16868))
* Bump pydantic from 2.5.3 to 2.6.0. ([\#16888](element-hq/synapse#16888))
* Bump sentry-sdk from 1.39.1 to 1.40.0. ([\#16889](element-hq/synapse#16889))
* Bump serde from 1.0.195 to 1.0.196. ([\#16867](element-hq/synapse#16867))
* Bump serde_json from 1.0.111 to 1.0.113. ([\#16866](element-hq/synapse#16866))
* Bump sigstore/cosign-installer from 3.3.0 to 3.4.0. ([\#16890](element-hq/synapse#16890))
* Bump types-pillow from 10.1.0.2 to 10.2.0.20240125. ([\#16864](element-hq/synapse#16864))
* Bump types-requests from 2.31.0.10 to 2.31.0.20240125. ([\#16886](element-hq/synapse#16886))
* Bump types-setuptools from 69.0.0.0 to 69.0.0.20240125. ([\#16863](element-hq/synapse#16863))

# Synapse 1.100.0 (2024-01-30)

No significant changes since 1.100.0rc3.




# Synapse 1.100.0rc3 (2024-01-24)

### Bugfixes

- Fix database performance regression due to changing Postgres table statistics. Introduced in v1.100.0rc1. ([\#16849](element-hq/synapse#16849))




# Synapse 1.100.0rc2 (2024-01-24)

This version is the same as 1.100.0rc1 but with fixes to the release process.

### Internal Changes

- Downgrade the `download-artifact` and `upload-artifact` actions to v3 due to breaking changes. ([\#16847](element-hq/synapse#16847))


# Synapse 1.100.0rc1 (2024-01-23)

*This version was never released to PyPI or the Debian repository due to failures in the automatic part of the release process.*

### Features

- Advertise experimental support for [MSC4028](matrix-org/matrix-spec-proposals#4028) through `/_matrix/clients/versions` if enabled. Contributed by @hanadi92. ([\#16787](element-hq/synapse#16787))

### Bugfixes

- Handle wildcard type filters properly for room messages endpoint. Contributed by Mo Balaa. ([\#14984](element-hq/synapse#14984))

### Improved Documentation

- Add a link to the "Request log format" explainer on the "Logging sample config" documentation page. ([\#16778](element-hq/synapse#16778))
- Fix broken links in issue templates and documentation. ([\#16810](element-hq/synapse#16810))
- NGINX listen http2 deprecation in documentation template for reverse proxy. ([\#16831](element-hq/synapse#16831))

### Internal Changes

- Faster partial join to room with complex auth graph. ([\#7](element-hq/synapse#7))
- Improve DB performance of calculating badge counts for push. ([\#16756](element-hq/synapse#16756))
- Split up deleting devices into batches. ([\#16766](element-hq/synapse#16766))
- Remove CI check for sign-off as we require a CLA signature instead. ([\#16776](element-hq/synapse#16776))
- Ensure CI fails when linting fails to make sure auto-merge does the correct thing. ([\#16781](element-hq/synapse#16781))
- Faster load recents for sync by reducing amount of state pulled out. ([\#16783](element-hq/synapse#16783))
- Reduce amount of state pulled out when querying federation hierachy. ([\#16785](element-hq/synapse#16785))
- Pull less state out of the DB when we retry fetching old events during backfill. ([\#16788](element-hq/synapse#16788))
- Optimize query for fetching to-device messages in `/sync`. ([\#16805](element-hq/synapse#16805))
- Reject OIDC config when `client_secret` isn't specified, but the auth method requires one. ([\#16806](element-hq/synapse#16806))
- Allow room creation but not publishing to continue if room publication rules are violated when creating
  a new room. ([\#16811](element-hq/synapse#16811))
- Bump minimum supported Rust version to 1.65.0. ([\#16818](element-hq/synapse#16818))
- Fixup copyright lines in file headers after the licensing change. ([\#16820](element-hq/synapse#16820))
- Add a `--generate-only` option to the internal configuration/launch script for Complement. ([\#16828](element-hq/synapse#16828))
- Preparatory work for tweaking performance of auth chain lookups. ([\#16833](element-hq/synapse#16833))
- Speed up e2e device keys queries for bot accounts. ([\#16841](element-hq/synapse#16841))

### Updates to locked dependencies

* Bump actions/cache from 3 to 4. ([\#16832](element-hq/synapse#16832))
* Bump actions/download-artifact from 3 to 4. ([\#16795](element-hq/synapse#16795))
* Bump actions/upload-artifact from 3 to 4. ([\#16796](element-hq/synapse#16796))
* Bump anyhow from 1.0.75 to 1.0.79. ([\#16789](element-hq/synapse#16789))
* Bump authlib from 1.2.1 to 1.3.0. ([\#16801](element-hq/synapse#16801))
* Bump dawidd6/action-download-artifact from 2.28.0 to 3.0.0. ([\#16794](element-hq/synapse#16794))
* Bump immutabledict from 4.0.0 to 4.1.0. ([\#16812](element-hq/synapse#16812))
* Bump isort from 5.13.1 to 5.13.2. ([\#16835](element-hq/synapse#16835))
* Bump lxml from 4.9.3 to 5.1.0. ([\#16813](element-hq/synapse#16813))
* Bump pillow from 10.1.0 to 10.2.0. ([\#16802](element-hq/synapse#16802))
* Bump pydantic from 2.5.2 to 2.5.3. ([\#16836](element-hq/synapse#16836))
* Bump pyo3 from 0.20.0 to 0.20.2. ([\#16791](element-hq/synapse#16791))
* Bump regex from 1.9.6 to 1.10.3. ([\#16837](element-hq/synapse#16837))
* Bump ruff from 0.1.13 to 0.1.14. ([\#16838](element-hq/synapse#16838))
* Bump ruff from 0.1.7 to 0.1.13. ([\#16814](element-hq/synapse#16814))
* Bump sentry-sdk from 1.35.0 to 1.39.1. ([\#16799](element-hq/synapse#16799))
* Bump serde_json from 1.0.108 to 1.0.111. ([\#16792](element-hq/synapse#16792))
* Bump service-identity from 23.1.0 to 24.1.0. ([\#16816](element-hq/synapse#16816))
* Bump types-commonmark from 0.9.2.4 to 0.9.2.20240106. ([\#16797](element-hq/synapse#16797))
* Bump types-jsonschema from 4.20.0.0 to 4.20.0.20240105. ([\#16800](element-hq/synapse#16800))
* Bump types-jsonschema from 4.20.0.20240105 to 4.21.0.20240118. ([\#16834](element-hq/synapse#16834))
* Bump types-netaddr from 0.9.0.1 to 0.10.0.20240106. ([\#16839](element-hq/synapse#16839))
* Bump typing-extensions from 4.8.0 to 4.9.0. ([\#16815](element-hq/synapse#16815))


# Synapse 1.99.0 (2024-01-16)

Synapse 1.99.0 is the first Synapse release under an AGPLv3.0 licence (with CLA to enable Element to sell AGPL
exceptions). You can read more about this here:

 - https://matrix.org/blog/2023/11/06/future-of-synapse-dendrite/
 - https://element.io/blog/element-to-adopt-agplv3/
 - https://element.io/blog/synapse-now-lives-at-github-com-element-hq-synapse/

No significant changes since 1.99.0rc1.


# Synapse 1.99.0rc1 (2024-01-09)

### Features

- Add [config options](https://element-hq.github.io/synapse/v1.99/usage/configuration/config_documentation.html#server_notices) to set the avatar and the topic of the server notices room, as well as the avatar of the server notices user. ([\#16679](matrix-org/synapse#16679))
- Add config option [`email.notif_delay_before_mail`](https://element-hq.github.io/synapse/v1.99/usage/configuration/config_documentation.html#email) to tweak the delay before an email is sent following a notification. ([\#16696](matrix-org/synapse#16696))
- Add new configuration option [`sentry.environment`](https://element-hq.github.io/synapse/v1.99/usage/configuration/config_documentation.html#sentry) for improved system monitoring. Contributed by @zeeshanrafiqrana. ([\#16738](matrix-org/synapse#16738))
- Filter out rooms from the room directory being served to other homeservers when those rooms block that homeserver by their Access Control Lists. ([\#16759](element-hq/synapse#16759))

### Bugfixes

- Fix a long-standing bug where the signing keys generated by Synapse were world-readable. Contributed by Fabian Klemp. ([\#16740](matrix-org/synapse#16740))
- Fix email verification redirection. Contributed by Fadhlan Ridhwanallah. ([\#16761](element-hq/synapse#16761))
- Fixed a bug that prevented users from being queried by display name if it contains non-ASCII characters. ([\#16767](element-hq/synapse#16767))
- Allow reactivate user without password with Admin API in some edge cases. ([\#16770](element-hq/synapse#16770))
- Adds the `recursion_depth` parameter to the response of the /relations endpoint if MSC3981 recursion is being performed. ([\#16775](element-hq/synapse#16775))

### Improved Documentation

- Added version picker for Synapse documentation. Contributed by @Dmytro27Ind. ([\#16533](matrix-org/synapse#16533))
- Clarify that `password_config.enabled: "only_for_reauth"` does not allow new logins to be created using password auth. ([\#16737](matrix-org/synapse#16737))
- Remove value from header in configuration documentation for `refresh_token_lifetime`. ([\#16763](element-hq/synapse#16763))
- Add another custom statistics collection server to the documentation. Contributed by @loelkes. ([\#16769](element-hq/synapse#16769))

### Internal Changes

- Remove run-once workflow after adding the version picker to the documentation. ([\#9453](element-hq/synapse#9453))
- Update the implementation of [MSC2965](matrix-org/matrix-spec-proposals#2965) (OIDC Provider discovery). ([\#16726](matrix-org/synapse#16726))
- Move the rust stubs inline for better IDE integration. ([\#16757](element-hq/synapse#16757))
- Fix sample config doc CI. ([\#16758](element-hq/synapse#16758))
- Simplify event internal metadata class. ([\#16762](element-hq/synapse#16762), [\#16780](element-hq/synapse#16780))
- Sign the published docker image using [cosign](https://docs.sigstore.dev/). ([\#16774](element-hq/synapse#16774))
- Port `EventInternalMetadata` class to Rust. ([\#16782](element-hq/synapse#16782))



### Updates to locked dependencies

* Bump actions/setup-go from 4 to 5. ([\#16749](matrix-org/synapse#16749))
* Bump actions/setup-python from 4 to 5. ([\#16748](matrix-org/synapse#16748))
* Bump immutabledict from 3.0.0 to 4.0.0. ([\#16743](matrix-org/synapse#16743))
* Bump isort from 5.12.0 to 5.13.0. ([\#16745](matrix-org/synapse#16745))
* Bump isort from 5.13.0 to 5.13.1. ([\#16752](matrix-org/synapse#16752))
* Bump pydantic from 2.5.1 to 2.5.2. ([\#16747](matrix-org/synapse#16747))
* Bump ruff from 0.1.6 to 0.1.7. ([\#16746](matrix-org/synapse#16746))
* Bump types-setuptools from 68.2.0.2 to 69.0.0.0. ([\#16744](matrix-org/synapse#16744))
anoadragon453 added a commit that referenced this pull request Mar 5, 2024
@anoadragon453
Revert "Improve DB performance of calculating badge counts for push. (#…
2b78ad3 
…16756)"

This reverts commit b11f7b5.
anoadragon453 added a commit that referenced this pull request Mar 5, 2024
@anoadragon453
Revert "Improve DB performance of calculating badge counts for push. (#…
8a05304 
…16756)" (#16979)
@anoadragon453 anoadragon453 mentioned this pull request Mar 5, 2024
Merged
netbsd-srcmastr pushed a commit to NetBSD/pkgsrc that referenced this pull request Mar 9, 2024
@Midar
Update chat/matrix-synapse to 1.102.0
904bbd9 
# Synapse 1.102.0 (2024-03-05)

### Bugfixes

- Revert element-hq/synapse#16756, which caused incorrect notification counts on mobile clients since v1.100.0. ([\#16979](element-hq/synapse#16979))


# Synapse 1.102.0rc1 (2024-02-20)

### Features

- A metric was added for emails sent by Synapse, broken down by type: `synapse_emails_sent_total`. Contributed by Remi Rampin. ([\#16881](element-hq/synapse#16881))

### Bugfixes

- Do not send multiple concurrent requests for keys for the same server. ([\#16894](element-hq/synapse#16894))
- Fix performance issue when joining very large rooms that can cause the server to lock up. Introduced in v1.100.0. ([\#16903](element-hq/synapse#16903))
- Always prefer unthreaded receipt when >1 exist ([MSC4102](matrix-org/matrix-spec-proposals#4102)). ([\#16927](element-hq/synapse#16927))

### Improved Documentation

- Fix a small typo in the Rooms section of the Admin API documentation. Contributed by @RainerZufall187. ([\#16857](element-hq/synapse#16857))

### Internal Changes

- Don't invalidate the entire event cache when we purge history. ([\#16905](element-hq/synapse#16905))
- Add experimental config option to not send device list updates for specific users. ([\#16909](element-hq/synapse#16909))
- Fix incorrect docker hub link in release script. ([\#16910](element-hq/synapse#16910))



### Updates to locked dependencies

* Bump attrs from 23.1.0 to 23.2.0. ([\#16899](element-hq/synapse#16899))
* Bump bcrypt from 4.0.1 to 4.1.2. ([\#16900](element-hq/synapse#16900))
* Bump pygithub from 2.1.1 to 2.2.0. ([\#16902](element-hq/synapse#16902))
* Bump sentry-sdk from 1.40.0 to 1.40.3. ([\#16898](element-hq/synapse#16898))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@anoadragon453 anoadragon453 anoadragon453 approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@erikjohnston @anoadragon453