Make rocksdb configurable #1658

evgeniy-scherbina · 2023-07-31T17:01:40Z

Related commit: Kava-Labs/tm-db@7fb308e

Steps before merge:

make tm-db release and change version in go.mod - DONE
remove temporary test.sh script - DONE

Future improvements:

somehow integrate CheckOptionsCompatibility
add e2e tests?

galxy25 · 2023-08-09T17:30:27Z

.github/workflows/ci-rocksdb-build.yml

@@ -17,6 +17,6 @@ jobs:
      - name: build rocksdb dependency
        run: bash ${GITHUB_WORKSPACE}/.github/scripts/install-rocksdb.sh
        env:
-          ROCKSDB_VERSION: v7.10.2
+          ROCKSDB_VERSION: v8.1.1


https://github.com/Kava-Labs/infrastructure/blob/master/packer/kava-ami/kava-internal-testnet-rocksdb.pkrvars.hcl is what internal testnet is using, will need to update the config and update existing nodes using the kava-node-updater(see https://kava-labs.atlassian.net/wiki/spaces/ENG/pages/1105231874/In+Place+Updates+of+a+Kava+Network and https://kava-labs.atlassian.net/wiki/spaces/ENG/pages/1147174913/How+to+upgrade+Kava+Mainnet+or+Testnets#Kava-Node-Updater-CLI for details - for everywhere this change needs to be deployed to (e.g. https://github.com/Kava-Labs/infrastructure/blob/master/packer/kava-ami/kava-public-testnet-rocksdb.pkrvars.hcl and https://github.com/Kava-Labs/infrastructure/blob/master/packer/kava-ami/kava-14-rocksdb.pkrvars.hcl) - doesn't need to be done in this pr but @evgeniy-scherbina please add a shortcut ticket to the epic for this work to track it

nddeluca

Looks good. We will have quite a few more options in the future, so we'll likely want to group by a prefix at some point for better organization & a unique namespace

nddeluca · 2023-08-10T15:58:29Z

cmd/kava/cmd/root.go

@@ -105,13 +106,15 @@ func addSubCmds(rootCmd *cobra.Command, encodingConfig params.EncodingConfig, de
 		encodingConfig: encodingConfig,
 	}

+	opts := ethermintserver.StartOptions{


options look good 👍

nddeluca · 2023-08-10T16:02:58Z

cmd/kava/opendb/opendb_rocksdb.go

+
+// overrideDBOpts merges dbOpts and appOpts, appOpts takes precedence
+func overrideDBOpts(dbOpts *grocksdb.Options, appOpts types.AppOptions) *grocksdb.Options {
+	maxOpenFiles := appOpts.Get(maxOpenFilesDBOptName)


These maybe should be prefixed so they can be grouped in app.toml? db.max_open_files, rocksdb.max_open_files, etc? Not sure the best prefix

rocksdb.max_open_files looks good

Make sure rocksdb tests are running in CI

This reverts commit 90fbe1a.

* Make rocksdb configurable * Make sure rocksdb tests are running in CI * Updating ci-rocksdb-build workflow * Remove test.sh * Update tm-db dependency (cherry picked from commit 90fbe1a) # Conflicts: # .github/workflows/ci-rocksdb-build.yml # Dockerfile-rocksdb # cmd/kava/cmd/root.go # go.mod # go.sum

* Make rocksdb configurable * Make sure rocksdb tests are running in CI * Updating ci-rocksdb-build workflow * Remove test.sh * Update tm-db dependency (cherry picked from commit 90fbe1a) # Conflicts: # Dockerfile-rocksdb # cmd/kava/cmd/root.go

* Make rocksdb configurable * Make sure rocksdb tests are running in CI * Updating ci-rocksdb-build workflow * Remove test.sh * Update tm-db dependency (cherry picked from commit 90fbe1a)

* Make rocksdb configurable (#1658) * Make rocksdb configurable * Make sure rocksdb tests are running in CI * Updating ci-rocksdb-build workflow * Remove test.sh * Update tm-db dependency (cherry picked from commit 90fbe1a) # Conflicts: # .github/workflows/ci-rocksdb-build.yml # Dockerfile-rocksdb # cmd/kava/cmd/root.go # go.mod # go.sum * Resolve conflicts after mergify * Rocksdb Metrics (#1692) * Rocksdb Metrics * Add rocksdb namespace for options * Adding help to the metrics * CR's fixes * CR's fixes * CR's fixes * Increase number of options to configure rocksdb (#1696) * Change ethermint dependency from local path to release tag --------- Co-authored-by: Evgeniy Scherbina <[email protected]>

* Make rocksdb configurable (#1658) * Make rocksdb configurable * Make sure rocksdb tests are running in CI * Updating ci-rocksdb-build workflow * Remove test.sh * Update tm-db dependency (cherry picked from commit 90fbe1a) # Conflicts: # .github/workflows/ci-rocksdb-build.yml # Dockerfile-rocksdb # cmd/kava/cmd/root.go # go.mod # go.sum * Resolve conflicts after mergify * Rocksdb Metrics (#1692) * Rocksdb Metrics * Add rocksdb namespace for options * Adding help to the metrics * CR's fixes * CR's fixes * CR's fixes * Increase number of options to configure rocksdb (#1696) * Change ethermint dependency to release tag --------- Co-authored-by: Evgeniy Scherbina <[email protected]>

* Make rocksdb configurable (#1658) * Make rocksdb configurable * Make sure rocksdb tests are running in CI * Updating ci-rocksdb-build workflow * Remove test.sh * Update tm-db dependency (cherry picked from commit 90fbe1a) # Conflicts: # Dockerfile-rocksdb # cmd/kava/cmd/root.go * Resolve conflicts after mergify * Rocksdb Metrics (#1692) * Rocksdb Metrics * Add rocksdb namespace for options * Adding help to the metrics * CR's fixes * CR's fixes * CR's fixes * Increase number of options to configure rocksdb (#1696) --------- Co-authored-by: Evgeniy Scherbina <[email protected]>

* Make rocksdb configurable (#1658) * Make rocksdb configurable * Make sure rocksdb tests are running in CI * Updating ci-rocksdb-build workflow * Remove test.sh * Update tm-db dependency (cherry picked from commit 90fbe1a) * Rocksdb Metrics (#1692) * Rocksdb Metrics * Add rocksdb namespace for options * Adding help to the metrics * CR's fixes * CR's fixes * CR's fixes * Increase number of options to configure rocksdb (#1696) --------- Co-authored-by: Evgeniy Scherbina <[email protected]>

evgeniy-scherbina force-pushed the yevhenii/rocksdb-config branch 8 times, most recently from dd0164b to 8673449 Compare August 4, 2023 17:34

evgeniy-scherbina requested a review from nddeluca August 4, 2023 17:35

evgeniy-scherbina marked this pull request as ready for review August 4, 2023 17:35

evgeniy-scherbina force-pushed the yevhenii/rocksdb-config branch 2 times, most recently from 8e910ff to 2e2bb72 Compare August 4, 2023 18:16

Make rocksdb configurable

6eb3072

evgeniy-scherbina force-pushed the yevhenii/rocksdb-config branch from 2e2bb72 to 6eb3072 Compare August 4, 2023 18:41

evgeniy-scherbina requested a review from galxy25 August 9, 2023 17:27

galxy25 reviewed Aug 9, 2023

View reviewed changes

galxy25 approved these changes Aug 9, 2023

View reviewed changes

nddeluca approved these changes Aug 10, 2023

View reviewed changes

pirtleshell approved these changes Aug 16, 2023

View reviewed changes

evgeniy-scherbina force-pushed the yevhenii/rocksdb-config branch from 2b2514e to 6eb3072 Compare August 21, 2023 20:20

evgeniy-scherbina added 2 commits August 21, 2023 16:45

Make sure rocksdb tests are running in CI

14ebb8c

Merge pull request #1663 from Kava-Labs/yevhenii/rocksdb-fix-tests

e9005c8

Make sure rocksdb tests are running in CI

evgeniy-scherbina force-pushed the yevhenii/rocksdb-config branch 2 times, most recently from 122c632 to 750082e Compare August 21, 2023 22:19

Updating ci-rocksdb-build workflow

ecb523f

evgeniy-scherbina force-pushed the yevhenii/rocksdb-config branch from 042d096 to ecb523f Compare August 22, 2023 01:03

evgeniy-scherbina added 2 commits August 21, 2023 21:16

Remove test.sh

7c99bd7

Update tm-db dependency

d005fca

evgeniy-scherbina merged commit 90fbe1a into master Aug 22, 2023
9 checks passed

evgeniy-scherbina deleted the yevhenii/rocksdb-config branch August 22, 2023 15:24

evgeniy-scherbina added a commit that referenced this pull request Aug 23, 2023

Revert "Make rocksdb configurable (#1658)"

a99a578

This reverts commit 90fbe1a.

evgeniy-scherbina added the A:backport/kava_2222-10 Backport patches to all kava_2222-10 release/* branches label Sep 5, 2023

This was referenced Sep 5, 2023

Make rocksdb configurable (backport #1658) #1697

Merged

Make rocksdb configurable (backport #1658) #1698

Merged

mergify bot mentioned this pull request Sep 5, 2023

Make rocksdb configurable (backport #1658) #1699

Merged

mergify bot mentioned this pull request Sep 5, 2023

Make rocksdb configurable (backport #1658) #1700

Merged

mergify bot mentioned this pull request Sep 5, 2023

Make rocksdb configurable (backport #1658) #1701

Merged

mergify bot mentioned this pull request Sep 5, 2023

Make rocksdb configurable (backport #1658) #1702

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make rocksdb configurable #1658

Make rocksdb configurable #1658

evgeniy-scherbina commented Jul 31, 2023 •

edited

Loading

galxy25 Aug 9, 2023

nddeluca left a comment

nddeluca Aug 10, 2023

nddeluca Aug 10, 2023

evgeniy-scherbina Aug 21, 2023

Make rocksdb configurable #1658

Make rocksdb configurable #1658

Conversation

evgeniy-scherbina commented Jul 31, 2023 • edited Loading

galxy25 Aug 9, 2023

Choose a reason for hiding this comment

nddeluca left a comment

Choose a reason for hiding this comment

nddeluca Aug 10, 2023

Choose a reason for hiding this comment

nddeluca Aug 10, 2023

Choose a reason for hiding this comment

evgeniy-scherbina Aug 21, 2023

Choose a reason for hiding this comment

evgeniy-scherbina commented Jul 31, 2023 •

edited

Loading