feat(rockdb): Allow customization of Rocksdb threads number

[ltagliamonte-dd]

Allow customization of rocksdb low and high thread number

[torwig]

@ltagliamonte-dd Could you please elaborate on the change? I see that we can set max_background_compactions and max_background_flushes. It's not enough? Or it's something different?

[ltagliamonte-dd]

@ltagliamonte-dd Could you please elaborate on the change? I see that we can set max_background_compactions and max_background_flushes. It's not enough? Or it's something different?

Hello @torwig, I'm very new to rocksdb, I may be wrong, but:
max_background_jobs/max_background_compactions/max_background_flushes are higher-level setting that controls the total number of concurrent background operations that RocksDB will perform.

SetBackgroundThreads controls the size of the thread pools separately for HIGH priority (typically compactions) and LOW priority (typically flushes) operations.

this is my understanding reading this doc

so for optimal perfomances you should configure both.

[git-hulk]

@ltagliamonte-dd You could just control the compaction/flush thread number by configuring:

• rocksdb.max_background_compactions (default is LOW priority)
• rocksdb.max_background_flushes (default is HIGH priority)

But those two parameters might be deprecated in future versions. It will use 1/4 * max_background_jobs as the flush thread number if both of them are not set. See GetBGJobLimits

[ltagliamonte-dd]

@git-hulk I may be reading this wrong, but where is it the correlation between this params (max_background_jobs/max_background_compactions/max_background_flushes) and the number of threads in rocksdb codebase?

In this example in the doc I see the setting of the threadpool size and max_background_jobs
https://github.com/facebook/rocksdb/wiki/Thread-Pool

[git-hulk]

where is it the correlation between this params (max_background_jobs/max_background_compactions/max_background_flushes) and the number of threads in rocksdb codebase?

The correlation, you could refer: GetBGJobLimits

[ltagliamonte-dd]

Closing in favor of using max_background_compactions and max_background_flushes has the same effect.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
62.5% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud