Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SeiDB+OCC for benchmark #351

Open
wants to merge 156 commits into
base: main
Choose a base branch
from
Open
Changes from 1 commit
Commits
Show all changes
156 commits
Select commit Hold shift + click to select a range
5e3f93f
Changes for memiavl
Jul 26, 2023
84bdda0
Fix changes for config
Jul 26, 2023
5a6e3eb
Remove debug log
Jul 26, 2023
539054b
fix go mod fir iavl
Jul 26, 2023
9d2e9c1
Bump golang version to 1.20
Jul 26, 2023
b717c11
Bump to 1.20
Jul 26, 2023
82df860
Fix go version
Jul 26, 2023
a9730a9
Expose Snapshot function
Jul 27, 2023
73eee20
Bump version
yzang2019 Aug 30, 2023
929da20
Bump mmap-iavl version
yzang2019 Sep 1, 2023
b03c2d4
Fix go sum
yzang2019 Sep 1, 2023
518544a
Add method for snapshot
yzang2019 Sep 2, 2023
2d06820
Bump iavl version
yzang2019 Sep 2, 2023
39068d8
Remove condition check
yzang2019 Sep 2, 2023
0d8aa92
Fix snapshot error handling logic
yzang2019 Sep 3, 2023
cd731c9
Merge branch 'main' into yzang/memiavl
yzang2019 Sep 3, 2023
6431ea7
Bump iavl and tendermint version
yzang2019 Sep 4, 2023
20c9714
Adding close for cache multistore in abci query
yzang2019 Sep 6, 2023
ed5e785
Add logs for cache multistore
yzang2019 Sep 7, 2023
6880b18
Add logs for cache multistore
yzang2019 Sep 7, 2023
ec9638a
Switch to seidb
yzang2019 Oct 11, 2023
6bc6c90
Add occ todos / comments (#317)
udpatil Sep 13, 2023
b66d23e
Multiversion Item Implementation and Tests (#318)
udpatil Sep 26, 2023
0048776
[occ] Add incarnation field (#321)
udpatil Sep 29, 2023
5d8941c
[occ] Implement basic multiversion store (#322)
udpatil Oct 6, 2023
dac5f7b
[occ] Add concurrency worker configuration (#324)
stevenlanders Oct 9, 2023
94bb98f
[occ] Occ multiversion store (#326)
udpatil Oct 10, 2023
5f89416
[occ] Add batch tx delivery interface (#327)
stevenlanders Oct 10, 2023
571d00a
[occ] MVKV store implementation and tests (#323)
udpatil Oct 10, 2023
9886602
[occ] Add validation function for transaction state to multiversionst…
udpatil Oct 13, 2023
293ac79
[occ] Add basic worker task and scheduler shell (#328)
stevenlanders Oct 17, 2023
dfb2260
[occ] Implement iterator for mvkv (#329)
udpatil Oct 17, 2023
663716a
fix dependency (#334)
udpatil Oct 17, 2023
b34d61c
[occ] Iterateset tracking and validation implementation (#337)
udpatil Oct 19, 2023
0aebbc9
[occ] Add scheduler logic for validation (#336)
stevenlanders Oct 19, 2023
096041b
[occ] Fix situation where no stores causes a panic (#338)
stevenlanders Oct 20, 2023
0b9193c
Add occ flag check to context (#340)
stevenlanders Oct 23, 2023
27484e4
[occ] Add struct field and helpers for estimate prefills (#341)
udpatil Oct 24, 2023
ab957f6
Add configs for seidb
yzang2019 Oct 24, 2023
7dcdd39
Bump seidb version
yzang2019 Oct 26, 2023
95ddc84
Fix map access panic (#343)
stevenlanders Oct 30, 2023
c51bdcb
Add query multi store for cosmos sdk
yzang2019 Nov 1, 2023
6302d64
Add qms
yzang2019 Nov 1, 2023
1d56ec8
Add snapshot manager for SC and SS
yzang2019 Nov 2, 2023
b96d79f
Fix snapshot manager
yzang2019 Nov 3, 2023
eab249d
Merge latest main
yzang2019 Nov 3, 2023
be4a4ae
Gen estimates writeset (#344)
udpatil Nov 3, 2023
948d27a
Bump seidb version
yzang2019 Nov 6, 2023
931e2f6
[OCC] Add trace spans to scheduler (#347)
stevenlanders Nov 6, 2023
2cc26a2
Add pruning settings for qms
yzang2019 Nov 7, 2023
9b57f8d
Revert change
yzang2019 Nov 7, 2023
cc31016
Add config for ss pruning manager
yzang2019 Nov 8, 2023
eac8657
[occ] Fix parent store readset validation (#348)
udpatil Nov 10, 2023
4111eeb
update lock to read lock on read
stevenlanders Nov 13, 2023
5bd9359
make validates parallel
stevenlanders Nov 13, 2023
09eaa06
add narrower lock
stevenlanders Nov 13, 2023
bf22e17
Bump seidb version
yzang2019 Nov 13, 2023
66ba9e6
Add config for num workers
yzang2019 Nov 13, 2023
06833bd
Add log for snapshot restore time
yzang2019 Nov 13, 2023
8818539
fix panic
stevenlanders Nov 14, 2023
1505649
make assertion dynamic
stevenlanders Nov 14, 2023
e321cf5
add more parallelization
stevenlanders Nov 14, 2023
b64efe1
add more parallelization
stevenlanders Nov 14, 2023
0d53d94
fix nested span
stevenlanders Nov 14, 2023
2752c9b
fix apphash
stevenlanders Nov 14, 2023
ce568cd
update spans
stevenlanders Nov 14, 2023
ac01db9
back the mapCacheBackend with a sync.Map
stevenlanders Nov 15, 2023
dc8e018
Revert "back the mapCacheBackend with a sync.Map"
stevenlanders Nov 15, 2023
ec87828
try early validate
stevenlanders Nov 15, 2023
9830c6c
add validation for previous tx dependency
stevenlanders Nov 15, 2023
2859512
fix span for validate
stevenlanders Nov 15, 2023
34a755a
remove mvkv mutex
udpatil Nov 13, 2023
205d5ca
remove some telemetry calls on heavily used functions
udpatil Nov 13, 2023
582e526
comment out rest of telemetry
udpatil Nov 13, 2023
7bab1f4
refactor MVS to use sync maps
udpatil Nov 14, 2023
e109f7e
remove log
udpatil Nov 14, 2023
a29338d
remove cachesize metric
udpatil Nov 14, 2023
a2ad48c
refactor cachekv mutex into map of mtxs
udpatil Nov 14, 2023
47fcbb1
remove event emit
udpatil Nov 15, 2023
33b1602
refactor cachekv for concurrent safe operation
udpatil Nov 15, 2023
ddeb662
remove telemetry from validateAll
udpatil Nov 15, 2023
925ebec
remove telemetry from runTX
udpatil Nov 15, 2023
4973f07
separate commitKVcache locking for getting vs writing to cache
udpatil Nov 15, 2023
4205725
avoid execute race
stevenlanders Nov 15, 2023
f7aa55d
remove lock from writeCache in commitKVStoreCache
udpatil Nov 15, 2023
d15a1aa
Merge branch 'main' into yzang/memiavl
yzang2019 Nov 15, 2023
9dda5d7
Bump seidb version
yzang2019 Nov 15, 2023
a7a9bb7
Fix merge conflict
yzang2019 Nov 15, 2023
887b943
Bump seidb version
yzang2019 Nov 15, 2023
e2a259a
cleanup
stevenlanders Nov 15, 2023
b0cb3ff
Fix seidb configs
yzang2019 Nov 15, 2023
b3783f5
Add log for DeliverTxBatch time
yzang2019 Nov 15, 2023
8da49cf
Fix logging
yzang2019 Nov 15, 2023
961ef16
fix data race
stevenlanders Nov 15, 2023
a3618f5
Bump seidb version
yzang2019 Nov 15, 2023
ebb8fe1
Bump version
yzang2019 Nov 15, 2023
62e2bf2
Remove some logs
yzang2019 Nov 15, 2023
5a6b268
refactor setWriteset to write keys concurrently
udpatil Nov 16, 2023
f83fcda
Try disable tracing for tx execution
yzang2019 Nov 16, 2023
2297520
Fix nil pointer
yzang2019 Nov 16, 2023
7a57d9f
remove mvkv mutex
udpatil Nov 13, 2023
3611e8f
remove some telemetry calls on heavily used functions
udpatil Nov 13, 2023
8ff1b86
comment out rest of telemetry
udpatil Nov 13, 2023
2bf854a
refactor MVS to use sync maps
udpatil Nov 14, 2023
be51212
remove log
udpatil Nov 14, 2023
4679312
remove cachesize metric
udpatil Nov 14, 2023
a56d9a8
refactor cachekv mutex into map of mtxs
udpatil Nov 14, 2023
d93219c
remove event emit
udpatil Nov 15, 2023
7fd83ac
refactor cachekv for concurrent safe operation
udpatil Nov 15, 2023
92c693d
remove telemetry from validateAll
udpatil Nov 15, 2023
bdbd3a6
remove telemetry from runTX
udpatil Nov 15, 2023
b619bae
separate commitKVcache locking for getting vs writing to cache
udpatil Nov 15, 2023
4876150
remove lock from writeCache in commitKVStoreCache
udpatil Nov 15, 2023
10a0106
try setting writeset concurrently
udpatil Nov 15, 2023
8b5cac1
add log
udpatil Nov 16, 2023
faa45e9
add log
udpatil Nov 16, 2023
961d567
add log
udpatil Nov 16, 2023
744428f
add log
udpatil Nov 16, 2023
247cac3
add log
udpatil Nov 16, 2023
d8fe722
add log
udpatil Nov 16, 2023
b7ffe7d
add logger
udpatil Nov 16, 2023
38380a8
add logger
udpatil Nov 16, 2023
2de779c
add logs
udpatil Nov 16, 2023
ad48a78
add logs
udpatil Nov 16, 2023
5397208
add logs
udpatil Nov 16, 2023
922837b
add logs
udpatil Nov 16, 2023
f616719
add logs
udpatil Nov 16, 2023
c48b8ad
add logs
udpatil Nov 16, 2023
a7571ae
bump go mod
udpatil Nov 16, 2023
de87d52
add logs
udpatil Nov 16, 2023
33e7242
add logs
udpatil Nov 16, 2023
dc36ad5
add logs
udpatil Nov 16, 2023
efdac0e
add logs
udpatil Nov 16, 2023
aa9a276
add clear iterateset
udpatil Nov 16, 2023
5d96863
Merge latest branch
yzang2019 Nov 16, 2023
f597fcd
Revert "add logs"
udpatil Nov 17, 2023
107fde3
Fix iterateset validation to no longer delete expectedKeys
udpatil Nov 20, 2023
6175308
Merge branch 'occ-optimization' into seidb+occ
yzang2019 Nov 20, 2023
ed65ad0
Fix bug
yzang2019 Nov 20, 2023
839d155
Revert "try setting writeset concurrently"
udpatil Nov 21, 2023
adf2e8b
Merge branch 'occ-optimization' into seidb+occ
yzang2019 Nov 21, 2023
07afd1d
[OCC] Use worker pool to limit number of goroutines (#355)
stevenlanders Nov 21, 2023
abddea4
[OCC] Fix deadlock (#356)
stevenlanders Nov 21, 2023
a0d2aa9
Test no-op deliver tx
yzang2019 Nov 28, 2023
81a029f
Add logging for occ latency
yzang2019 Nov 28, 2023
3bae5dd
Add config to disable seqno (#385)
philipsu522 Jan 9, 2024
20f8928
Add migration handler for disabling seqno (#394)
philipsu522 Jan 11, 2024
3483ce7
Pick part1
yzang2019 Dec 27, 2023
87493f6
Part 2
yzang2019 Dec 27, 2023
5631500
Part 3
yzang2019 Dec 27, 2023
640567f
Part 4
yzang2019 Dec 28, 2023
16a5b68
Fix
yzang2019 Feb 5, 2024
c320a02
Fix
yzang2019 Feb 5, 2024
0aa8a5b
Fix
yzang2019 Feb 5, 2024
1972318
Add
yzang2019 Feb 5, 2024
5be6b78
Add log
yzang2019 Feb 15, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Part 4
yzang2019 committed Feb 5, 2024
commit 640567f1142568994e7632743c501066283bf0e7
8 changes: 1 addition & 7 deletions baseapp/options.go
Original file line number Diff line number Diff line change
@@ -4,13 +4,11 @@ import (
"fmt"
"io"

dbm "github.com/tendermint/tm-db"

"github.com/cosmos/cosmos-sdk/codec/types"
"github.com/cosmos/cosmos-sdk/snapshots"
"github.com/cosmos/cosmos-sdk/store"
sdk "github.com/cosmos/cosmos-sdk/types"
"github.com/cosmos/iavl"
dbm "github.com/tendermint/tm-db"
)

// File for storing in-package BaseApp optional functions,
@@ -41,10 +39,6 @@ func SetHaltTime(haltTime uint64) func(*BaseApp) {
return func(bapp *BaseApp) { bapp.setHaltTime(haltTime) }
}

func SetOrphanConfig(opts *iavl.Options) func(*BaseApp) {
return func(bapp *BaseApp) {}
}

// SetMinRetainBlocks returns a BaseApp option function that sets the minimum
// block retention height value when determining which heights to prune during
// ABCI Commit.
112 changes: 109 additions & 3 deletions go.mod
Original file line number Diff line number Diff line change
@@ -57,15 +57,118 @@ require (
go.opentelemetry.io/otel/trace v1.9.0
golang.org/x/crypto v0.14.0
golang.org/x/exp v0.0.0-20230811145659-89c5cff77bcb
golang.org/x/sync v0.4.0
google.golang.org/genproto/googleapis/api v0.0.0-20230920204549-e6e6cdab5c13
google.golang.org/grpc v1.58.3
google.golang.org/protobuf v1.31.0
gopkg.in/yaml.v2 v2.4.0
gotest.tools v2.2.0+incompatible
)


require (
filippo.io/edwards25519 v1.0.0-rc.1 // indirect
github.com/DataDog/zstd v1.4.5 // indirect
github.com/alitto/pond v1.8.3 // indirect
github.com/beorn7/perks v1.0.1 // indirect
github.com/cespare/xxhash v1.1.0 // indirect
github.com/cespare/xxhash/v2 v2.2.0 // indirect
github.com/cockroachdb/errors v1.8.1 // indirect
github.com/cockroachdb/logtags v0.0.0-20190617123548-eb05cc24525f // indirect
github.com/cockroachdb/pebble v0.0.0-20230819001538-1798fbf5956c // indirect
github.com/cockroachdb/redact v1.0.8 // indirect
github.com/cockroachdb/sentry-go v0.6.1-cockroachdb.2 // indirect
github.com/cockroachdb/tokenbucket v0.0.0-20230807174530-cc333fc44b06 // indirect
github.com/cosmos/gorocksdb v1.2.0 // indirect
github.com/creachadair/taskgroup v0.3.2 // indirect
github.com/danieljoos/wincred v1.0.2 // indirect
github.com/davecgh/go-spew v1.1.1 // indirect
github.com/desertbit/timer v0.0.0-20180107155436-c41aec40b27f // indirect
github.com/dgraph-io/badger/v3 v3.2103.2 // indirect
github.com/dgraph-io/ristretto v0.1.1 // indirect
github.com/dustin/go-humanize v1.0.1 // indirect
github.com/dvsekhvalnov/jose2go v0.0.0-20200901110807-248326c1351b // indirect
github.com/felixge/httpsnoop v1.0.1 // indirect
github.com/fsnotify/fsnotify v1.6.0 // indirect
github.com/go-kit/kit v0.12.0 // indirect
github.com/go-kit/log v0.2.1 // indirect
github.com/go-logfmt/logfmt v0.5.1 // indirect
github.com/go-logr/logr v1.2.3 // indirect
github.com/go-logr/stdr v1.2.2 // indirect
github.com/godbus/dbus v0.0.0-20190726142602-4481cbc300e2 // indirect
github.com/golang/glog v1.1.0 // indirect
github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da // indirect
github.com/golang/snappy v0.0.4 // indirect
github.com/google/flatbuffers v1.12.1 // indirect
github.com/google/go-cmp v0.5.9 // indirect
github.com/google/orderedcode v0.0.1 // indirect
github.com/google/uuid v1.3.0 // indirect
github.com/gorilla/websocket v1.5.0 // indirect
github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0 // indirect
github.com/gsterjov/go-libsecret v0.0.0-20161001094733-a6f4afe4910c // indirect
github.com/hashicorp/go-immutable-radix v1.3.1 // indirect
github.com/hashicorp/golang-lru v0.5.5-0.20210104140557-80c98217689d // indirect
github.com/hashicorp/hcl v1.0.0 // indirect
github.com/inconshreveable/mousetrap v1.0.1 // indirect
github.com/jmhodges/levigo v1.0.0 // indirect
github.com/kballard/go-shellquote v0.0.0-20180428030007-95032a82bc51 // indirect
github.com/keybase/go-keychain v0.0.0-20190712205309-48d3d31d256d // indirect
github.com/klauspost/compress v1.16.3 // indirect
github.com/kr/pretty v0.3.1 // indirect
github.com/kr/text v0.2.0 // indirect
github.com/ledgerwatch/erigon-lib v0.0.0-20230210071639-db0e7ed11263 // indirect
github.com/lib/pq v1.10.6 // indirect
github.com/libp2p/go-buffer-pool v0.0.2 // indirect
github.com/linxGnu/grocksdb v1.8.4 // indirect
github.com/mattn/go-colorable v0.1.13 // indirect
github.com/matttproud/golang_protobuf_extensions v1.0.4 // indirect
github.com/mitchellh/mapstructure v1.5.0 // indirect
github.com/mtibben/percent v0.2.1 // indirect
github.com/oasisprotocol/curve25519-voi v0.0.0-20210609091139-0a56a4bca00b // indirect
github.com/pelletier/go-toml v1.9.5 // indirect
github.com/pelletier/go-toml/v2 v2.0.5 // indirect
github.com/petermattis/goid v0.0.0-20230317030725-371a4b8eda08 // indirect
github.com/pmezard/go-difflib v1.0.0 // indirect
github.com/prometheus/client_model v0.3.0 // indirect
github.com/prometheus/procfs v0.9.0 // indirect
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec // indirect
github.com/rogpeppe/go-internal v1.11.0 // indirect
github.com/rs/cors v1.8.2 // indirect
github.com/sasha-s/go-deadlock v0.3.1 // indirect
github.com/spf13/afero v1.8.2 // indirect
github.com/spf13/jwalterweatherman v1.1.0 // indirect
github.com/subosito/gotenv v1.4.1 // indirect
github.com/tidwall/gjson v1.10.2 // indirect
github.com/tidwall/match v1.1.1 // indirect
github.com/tidwall/pretty v1.2.0 // indirect
github.com/tidwall/tinylru v1.1.0 // indirect
github.com/tidwall/wal v1.1.7 // indirect
github.com/zbiljic/go-filelock v0.0.0-20170914061330-1dbf7103ab7d // indirect
github.com/zondax/hid v0.9.1 // indirect
github.com/zondax/ledger-go v0.14.1 // indirect
go.etcd.io/bbolt v1.3.7 // indirect
go.opencensus.io v0.23.0 // indirect
golang.org/x/mod v0.11.0 // indirect
golang.org/x/net v0.17.0 // indirect
golang.org/x/sync v0.4.0 // indirect
golang.org/x/sys v0.13.0 // indirect
golang.org/x/term v0.13.0 // indirect
golang.org/x/text v0.13.0 // indirect
golang.org/x/tools v0.6.0 // indirect
google.golang.org/genproto v0.0.0-20230913181813-007df8e322eb // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20230920183334-c177e329c48b // indirect
gopkg.in/ini.v1 v1.67.0 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
lukechampine.com/uint128 v1.2.0 // indirect
modernc.org/cc/v3 v3.40.0 // indirect
modernc.org/ccgo/v3 v3.16.13 // indirect
modernc.org/libc v1.24.1 // indirect
modernc.org/mathutil v1.5.0 // indirect
modernc.org/memory v1.6.0 // indirect
modernc.org/opt v0.1.3 // indirect
modernc.org/sqlite v1.26.0 // indirect
modernc.org/strutil v1.1.3 // indirect
modernc.org/token v1.0.1 // indirect
nhooyr.io/websocket v1.8.6 // indirect
)

replace (
github.com/99designs/keyring => github.com/cosmos/keyring v1.1.7-0.20210622111912-ef00f8ac3d76
@@ -75,8 +178,11 @@ replace (
// TODO Remove it: https://github.com/cosmos/cosmos-sdk/issues/10409
github.com/gin-gonic/gin => github.com/gin-gonic/gin v1.7.0
github.com/gogo/protobuf => github.com/regen-network/protobuf v1.3.3-alpha.regen.1

github.com/sei-protocol/sei-db => github.com/sei-protocol/sei-db v0.0.30
github.com/tendermint/tendermint => github.com/sei-protocol/sei-tendermint v0.2.28
// Latest goleveldb is broken, we have to stick to this version
github.com/syndtr/goleveldb => github.com/syndtr/goleveldb v1.0.1-0.20210819022825-2ae1ddf74ef7
github.com/tendermint/tendermint => github.com/sei-protocol/sei-tendermint v0.2.37
// latest grpc doesn't work with with our modified proto compiler, so we need to enforce
// the following version across all dependencies.
google.golang.org/grpc => google.golang.org/grpc v1.33.2
676 changes: 648 additions & 28 deletions go.sum

Large diffs are not rendered by default.

7 changes: 0 additions & 7 deletions simapp/simd/cmd/root.go
Original file line number Diff line number Diff line change
@@ -7,7 +7,6 @@ import (
"path/filepath"

serverconfig "github.com/cosmos/cosmos-sdk/server/config"
"github.com/cosmos/iavl"
"github.com/spf13/cast"
"github.com/spf13/cobra"
tmmain "github.com/tendermint/tendermint/cmd/tendermint/commands"
@@ -306,12 +305,6 @@ func (a appCreator) newApp(logger log.Logger, db dbm.DB, traceStore io.Writer, t
baseapp.SetIAVLCacheSize(cast.ToInt(appOpts.Get(server.FlagIAVLCacheSize))),
baseapp.SetIAVLDisableFastNode(cast.ToBool(appOpts.Get(server.FlagIAVLFastNode))),
baseapp.SetCompactionInterval(cast.ToUint64(appOpts.Get(server.FlagCompactionInterval))),
baseapp.SetOrphanConfig(&iavl.Options{
SeparateOrphanStorage: cast.ToBool(appOpts.Get(server.FlagSeparateOrphanStorage)),
SeparateOphanVersionsToKeep: cast.ToInt64(appOpts.Get(server.FlagSeparateOrphanVersionsToKeep)),
NumOrphansPerFile: cast.ToInt(appOpts.Get(server.FlagNumOrphanPerFile)),
OrphanDirectory: cast.ToString(appOpts.Get(server.FlagOrphanDirectory)),
}),
)
}