Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

shim layer for reprocessing blocks #705

Draft
wants to merge 310 commits into
base: master
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
310 commits
Select commit Hold shift + click to select a range
f29efd6
try
darioush Dec 19, 2024
7f75d1e
try
darioush Dec 19, 2024
5ebdd58
oops
darioush Dec 19, 2024
a31dfb0
try
darioush Dec 19, 2024
69b9f59
try
darioush Dec 19, 2024
426baed
modify log
darioush Dec 19, 2024
8ea65e7
add log
darioush Dec 19, 2024
bbf32ce
try
darioush Dec 19, 2024
4361cfb
add hash
darioush Dec 19, 2024
1915247
try
darioush Dec 19, 2024
a306b81
try
darioush Dec 19, 2024
ce5b773
try
darioush Dec 19, 2024
514042a
try
darioush Dec 19, 2024
4745972
try
darioush Dec 19, 2024
515ce06
try
darioush Dec 19, 2024
6fea3bb
add logging for atomic tx
darioush Dec 19, 2024
9a940ba
use term string
darioush Dec 19, 2024
ac5230e
fix UTs
darioush Dec 19, 2024
a955bb8
fix UT again
darioush Dec 19, 2024
9d3a9a4
add snapshot iteration as test
darioush Dec 19, 2024
6369319
add merkledb flags
darioush Dec 19, 2024
30105d4
change units
darioush Dec 19, 2024
1ee79b5
fix options
darioush Dec 19, 2024
74f8e5b
support upgrades
darioush Dec 19, 2024
8237a65
support skipUpgradeCheck
darioush Dec 19, 2024
021fc39
decouple prefetcher from snap
darioush Dec 20, 2024
003ac7d
add usePersistedStartBlock
darioush Dec 20, 2024
436cae0
add tapeRecorder
darioush Dec 24, 2024
e414e58
better logging w/ tape
darioush Dec 24, 2024
fca800d
reduce logs
darioush Dec 24, 2024
02e97db
formatting
darioush Dec 24, 2024
9a4a636
fixes
darioush Dec 24, 2024
a21a86e
add file manager
darioush Dec 24, 2024
718b2a2
fix
darioush Dec 24, 2024
e277e99
add create permissions
darioush Dec 24, 2024
ee1d9bc
align files
darioush Dec 24, 2024
5ca3b36
dont cap on pathdb
darioush Dec 24, 2024
7172a14
add option for trie clean cache
darioush Dec 24, 2024
08d1cef
use tape representation
darioush Dec 25, 2024
a0bb20d
add verification for tx end markers
darioush Dec 25, 2024
52d9c3d
some postprocessing
darioush Dec 27, 2024
f704a65
formatting
darioush Dec 27, 2024
f38d9d5
formatting
darioush Dec 27, 2024
ee5080c
try
darioush Dec 27, 2024
48e37e0
update lastfile
darioush Dec 27, 2024
0f870d9
require again
darioush Dec 27, 2024
79f5265
measure state size
darioush Dec 27, 2024
5b1b6c1
fix
darioush Dec 27, 2024
d6562f1
term str
darioush Dec 27, 2024
c0dd1d9
try
darioush Dec 27, 2024
dfac973
try
darioush Dec 27, 2024
63135e7
try
darioush Dec 27, 2024
3643deb
move legend
darioush Dec 27, 2024
2c6097c
oops
darioush Dec 27, 2024
32a97be
nits
darioush Dec 27, 2024
5081cdd
fix
darioush Dec 27, 2024
c6c3c69
DrainAcceptorQueue
darioush Dec 27, 2024
9fee652
try read cache
darioush Dec 28, 2024
fb9870b
fix
darioush Dec 28, 2024
6296e0c
try
darioush Dec 28, 2024
77609e5
only count first access
darioush Dec 28, 2024
9ad00ba
no need for legend
darioush Dec 28, 2024
b0ce199
try
darioush Dec 28, 2024
4e5c90d
less allocs
darioush Dec 28, 2024
abd6d17
log % of state
darioush Dec 28, 2024
42de2e0
add details
darioush Dec 28, 2024
3f752f2
cmp fastcache
darioush Dec 28, 2024
8aac7d8
add cache delete
darioush Dec 28, 2024
4d1eb47
try otter
darioush Dec 28, 2024
3cda736
reduce logs
darioush Dec 28, 2024
bc1899a
try to figure out write recency
darioush Dec 28, 2024
5a5ab62
keep deletes
darioush Dec 28, 2024
31ca4bb
count 0 -> 0 as "update" (vs deletion or addition
darioush Dec 28, 2024
3969977
more logging
darioush Dec 28, 2024
678ba40
stats
darioush Dec 28, 2024
25d91ef
per tx
darioush Dec 28, 2024
22c1190
log
darioush Dec 28, 2024
280ecc0
try hst
darioush Dec 28, 2024
f0d5a66
try
darioush Dec 28, 2024
edc8f28
improve log
darioush Dec 28, 2024
44b2440
try
darioush Dec 28, 2024
b05e69b
try
darioush Dec 28, 2024
387a327
two hist
darioush Dec 28, 2024
4fdcbac
oops
darioush Dec 28, 2024
2f57196
bump inf
darioush Dec 28, 2024
b3028ae
try fix
darioush Dec 28, 2024
7b221ae
formatting
darioush Dec 28, 2024
01523a8
fix
darioush Dec 28, 2024
cbdb568
fix
darioush Dec 28, 2024
ea6b5bd
try
darioush Dec 28, 2024
5718b88
try bufio
darioush Dec 31, 2024
c7a4569
try
darioush Dec 31, 2024
b5616d1
try
darioush Dec 31, 2024
8590c8e
try
darioush Dec 31, 2024
67acfe0
try again
darioush Dec 31, 2024
5ee4152
try
darioush Jan 1, 2025
605aec2
try
darioush Jan 1, 2025
d4f7f9c
try
darioush Jan 1, 2025
69908df
try
darioush Jan 1, 2025
fd758a9
try
darioush Jan 1, 2025
41678e1
basic ipc backend
darioush Jan 2, 2025
cc48c83
remove verbose logs
darioush Jan 2, 2025
0c3425d
logEach for reprocess
darioush Jan 2, 2025
92e4e2c
change
darioush Jan 2, 2025
166b528
add writes from cache evicts
darioush Jan 3, 2025
4efc8c8
try
darioush Jan 3, 2025
40d9078
fix
darioush Jan 3, 2025
cdec960
track metadata
darioush Jan 3, 2025
e2fd0a5
fix start
darioush Jan 3, 2025
f837646
skip blocks
darioush Jan 3, 2025
ff6fc88
try
darioush Jan 3, 2025
666d506
try
darioush Jan 3, 2025
5edea85
improve skip log
darioush Jan 3, 2025
0f96133
try
darioush Jan 3, 2025
cb7bdb5
fix
darioush Jan 3, 2025
235a531
try
darioush Jan 3, 2025
e87059f
add TestQueryBlock
darioush Jan 3, 2025
7849e57
try
darioush Jan 3, 2025
61121ad
try
darioush Jan 3, 2025
9fb65dc
try
darioush Jan 3, 2025
6c99c19
try
darioush Jan 3, 2025
10a2556
Merge branch 'master' of github.com:ava-labs/coreth into prefetch-state
darioush Jan 3, 2025
afce6e4
fix
darioush Jan 3, 2025
03838e2
check genesis
darioush Jan 3, 2025
9c3c69a
verbose
darioush Jan 3, 2025
1655aa9
try
darioush Jan 3, 2025
12bed94
try
darioush Jan 3, 2025
860b590
log genesis
darioush Jan 3, 2025
c3cacaa
try
darioush Jan 3, 2025
4b918c5
too verbose
darioush Jan 3, 2025
59c726c
nit
darioush Jan 3, 2025
61fcaa7
patch
darioush Jan 3, 2025
849269f
only send unique updates to underlying storage
darioush Jan 3, 2025
73dc9ce
formatting
darioush Jan 3, 2025
0c5f2b4
ms
darioush Jan 3, 2025
95a2bfb
update formatting for negative state growth
darioush Jan 3, 2025
eb5df37
allow creartion of continuable db through pp
darioush Jan 3, 2025
13564f4
handle genesis + write batch
darioush Jan 3, 2025
3769b9b
try
darioush Jan 3, 2025
f2bf587
try
darioush Jan 3, 2025
0094036
try
darioush Jan 3, 2025
0f1e164
try
darioush Jan 3, 2025
e230478
snapshot
darioush Jan 3, 2025
b9ae451
Merge branch 'master' of github.com:ava-labs/coreth into prefetch-state
darioush Jan 3, 2025
ec47722
add code export support
darioush Jan 3, 2025
b655494
detect root divergance
darioush Jan 3, 2025
51b20ff
add log
darioush Jan 3, 2025
f6ff6f1
add iteration
darioush Jan 3, 2025
8498356
try
darioush Jan 3, 2025
73ea7e6
try
darioush Jan 3, 2025
0ff0a5b
try
darioush Jan 3, 2025
a006119
try
darioush Jan 3, 2025
47a4176
try
darioush Jan 3, 2025
f90888d
try
darioush Jan 3, 2025
946afa9
try
darioush Jan 3, 2025
c1fcee6
oops
darioush Jan 4, 2025
e158f03
verify expectation
darioush Jan 4, 2025
aa78bdb
check
darioush Jan 4, 2025
470c707
try
darioush Jan 4, 2025
e2b6679
try
darioush Jan 4, 2025
fe552dd
try
darioush Jan 4, 2025
fb80d8a
remove log
darioush Jan 4, 2025
46e0d05
try handle deletes
darioush Jan 4, 2025
078a355
preserve order
darioush Jan 4, 2025
c8e9a7b
try
darioush Jan 4, 2025
013c23a
try
darioush Jan 4, 2025
ad035d3
try
darioush Jan 4, 2025
4338bdc
try
darioush Jan 4, 2025
9a8d485
fix
darioush Jan 4, 2025
ad58a06
log ethdb prefix
darioush Jan 4, 2025
ebc817e
try
darioush Jan 4, 2025
ea59141
fix
darioush Jan 4, 2025
e4cae09
try
darioush Jan 4, 2025
d63d2b9
```oops
darioush Jan 4, 2025
7b0efa8
try
darioush Jan 4, 2025
975b747
try
darioush Jan 4, 2025
2a60161
try
darioush Jan 4, 2025
281139c
try storage befor accounts
darioush Jan 4, 2025
cbe40c2
undo logs
darioush Jan 4, 2025
6f0e942
remove logs
darioush Jan 4, 2025
c0d0b14
add option to continue from hash db
darioush Jan 7, 2025
b579c23
prefix option
darioush Jan 7, 2025
e248b94
try
darioush Jan 7, 2025
7618b6c
try
darioush Jan 7, 2025
5b60dd6
fix
darioush Jan 7, 2025
d6664be
try fixing storage delete
darioush Jan 8, 2025
f03ac77
write slim rlp snapshot
darioush Jan 9, 2025
d604e23
fix snapshot delete handling
darioush Jan 9, 2025
daaa145
pass-through node It
darioush Jan 10, 2025
5e3b97e
add snapshot del log
darioush Jan 10, 2025
235d1c1
nit
darioush Jan 10, 2025
8f880d5
track leaf dels
darioush Jan 10, 2025
d729a6b
add more logs
darioush Jan 10, 2025
b792779
nit
darioush Jan 10, 2025
309e453
try
darioush Jan 10, 2025
d8313bc
update iface
darioush Jan 10, 2025
316c546
add generated file just in case
darioush Jan 10, 2025
1d9f524
fix
darioush Jan 10, 2025
7645f6e
try
darioush Jan 10, 2025
93f50a0
fix comment
darioush Jan 10, 2025
403430d
clarify comment
darioush Jan 10, 2025
5c5eac8
patch empty values for storage tries
darioush Jan 10, 2025
269e701
try
darioush Jan 10, 2025
80ec4e2
more verbose
darioush Jan 10, 2025
aee1e17
fix
darioush Jan 10, 2025
0de3db2
add acc data
darioush Jan 10, 2025
99484c3
try
darioush Jan 10, 2025
a91f731
try
darioush Jan 10, 2025
2a407a5
try
darioush Jan 10, 2025
35fdc09
try
darioush Jan 10, 2025
eb61f69
undo
darioush Jan 10, 2025
21396aa
try
darioush Jan 10, 2025
2610e0f
undo
darioush Jan 10, 2025
232e74f
try
darioush Jan 10, 2025
58a7937
try
darioush Jan 10, 2025
23b5916
get val
darioush Jan 10, 2025
1649262
try
darioush Jan 10, 2025
60f7913
add log
darioush Jan 10, 2025
8a1bb22
fix
darioush Jan 10, 2025
ddb2b6a
Merge branch 'prefetch-state' of github.com:ava-labs/coreth into pref…
darioush Jan 10, 2025
2e5952a
undo logging
darioush Jan 10, 2025
0c548b4
apply logEach to tapeDir mode as well
darioush Jan 13, 2025
63f9293
oops
darioush Jan 13, 2025
35eec68
trackDeletedTries
darioush Jan 13, 2025
84ea934
only track actual deletes
darioush Jan 13, 2025
6a9d0c7
log deleted tries
darioush Jan 14, 2025
023897e
add more logging
darioush Jan 14, 2025
0746e7f
fix
darioush Jan 14, 2025
bb1fcb9
wspace
darioush Jan 14, 2025
b095d89
add prefix delete in post_processing
darioush Jan 14, 2025
e6ae493
log nit
darioush Jan 14, 2025
e3f6ac1
only need to track non-empty
darioush Jan 14, 2025
bb9b6ba
use prefixdelete in opcode selfdestruct
darioush Jan 14, 2025
543a7df
Revert "only need to track non-empty"
darioush Jan 14, 2025
4a1b8d5
add firewood backend
darioush Jan 15, 2025
511ee07
Merge branch 'master' of github.com:ava-labs/coreth into prefetch-state
darioush Jan 15, 2025
419884d
remove test db
darioush Jan 15, 2025
f6cebac
merge fixes
darioush Jan 15, 2025
efcbce3
handle empty batches
darioush Jan 15, 2025
9144240
add vm option
darioush Jan 16, 2025
5865517
add log timer
darioush Jan 16, 2025
133618c
reduce logs
darioush Jan 16, 2025
4f37033
only verify root if there's no write cache
darioush Jan 21, 2025
27cd797
try
darioush Jan 21, 2025
f50879b
try
darioush Jan 21, 2025
7d8a3af
try
darioush Jan 21, 2025
04791e3
ensure no loop in legacy
darioush Jan 22, 2025
01c7772
fix empty in geth
darioush Jan 22, 2025
79a46de
add histogram for snap val lens
darioush Jan 23, 2025
20c198b
dedup
darioush Jan 26, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 14 additions & 10 deletions core/block_validator.go
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,7 @@ import (
"github.com/ava-labs/coreth/core/types"
"github.com/ava-labs/coreth/params"
"github.com/ava-labs/coreth/trie"
"github.com/ethereum/go-ethereum/common"
)

// BlockValidator is responsible for validating block headers, uncles and
Expand All @@ -45,14 +46,17 @@ type BlockValidator struct {
config *params.ChainConfig // Chain configuration options
bc *BlockChain // Canonical block chain
engine consensus.Engine // Consensus engine used for validating

CheckRoot func(expected, got common.Hash) bool
}

// NewBlockValidator returns a new block validator which is safe for re-use
func NewBlockValidator(config *params.ChainConfig, blockchain *BlockChain, engine consensus.Engine) *BlockValidator {
validator := &BlockValidator{
config: config,
engine: engine,
bc: blockchain,
config: config,
engine: engine,
bc: blockchain,
CheckRoot: func(expected, got common.Hash) bool { return expected == got },
}
return validator
}
Expand Down Expand Up @@ -106,12 +110,12 @@ func (v *BlockValidator) ValidateBody(block *types.Block) error {
}

// Ancestor block must be known.
if !v.bc.HasBlockAndState(block.ParentHash(), block.NumberU64()-1) {
if !v.bc.HasBlock(block.ParentHash(), block.NumberU64()-1) {
return consensus.ErrUnknownAncestor
}
return consensus.ErrPrunedAncestor
}
// if !v.bc.HasBlockAndState(block.ParentHash(), block.NumberU64()-1) {
// if !v.bc.HasBlock(block.ParentHash(), block.NumberU64()-1) {
// return consensus.ErrUnknownAncestor
// }
// return consensus.ErrPrunedAncestor
// }
return nil
}

Expand All @@ -135,7 +139,7 @@ func (v *BlockValidator) ValidateState(block *types.Block, statedb *state.StateD
}
// Validate the state root against the received state root and throw
// an error if they don't match.
if root := statedb.IntermediateRoot(v.config.IsEIP158(header.Number)); header.Root != root {
if root := statedb.IntermediateRoot(v.config.IsEIP158(header.Number)); !v.CheckRoot(header.Root, root) {
return fmt.Errorf("invalid merkle root (remote: %x local: %x) dberr: %w", header.Root, root, statedb.Error())
}
return nil
Expand Down
Loading