-
Notifications
You must be signed in to change notification settings - Fork 183
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
feat(sync-service): Reduce memory use by hibernating idle shape proce…
…sses (#2240) Potential fix for #2229 Electric's memory footprint can get very large due to shape processes holding on to memory even when idle. This PR hibernates idle shape processes after 30 seconds of inactivity which garbage collects unused memory. This first graph shows 10MB changes coming in for 100 shapes (2 shapes per second). You can see this taking up 2000MB of memory which is not released even though the memory is no longer needed (it's been written to disk). <img width="679" alt="Screenshot 2025-01-22 at 12 06 41" src="https://github.com/user-attachments/assets/ae0d5da9-731f-4a9b-9fc1-3f47fa2ffab4" /> Not only is the memory not released, but the memory is amplified. In this example it's amplified by 2x since 2000MB is twice the 1000MB needed (10MB * 100 shapes). This is due to a binary copy that happens when CubDB persists data to file. Amplification can increase further do to having a binary version of the change as well as an Elixir term version of the change which is used for where clause filtering. This second graph is with `ELECTRIC_SHAPE_HIBERNATE_AFTER` set to `50ms`. A value this short is unlikely to be used in production but allows the effect to be seen quickly for the purposes of this benchmark. <img width="667" alt="Screenshot 2025-01-22 at 12 09 38" src="https://github.com/user-attachments/assets/5c22fa32-5e83-49aa-ab2b-d233a5af9c4d" /> As you can see, memory use flatlines at 120MB with hibernation rather than blowing up to 2000MB without hibernation.
- Loading branch information
Showing
7 changed files
with
31 additions
and
5 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,5 @@ | ||
--- | ||
"@core/sync-service": patch | ||
--- | ||
|
||
Reduce memory footprint by hibernating idle shapes |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters