Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[e3] FindBlockNum(...): seems broken TxNum value #12009

Open
wmitsuda opened this issue Sep 16, 2024 · 4 comments
Open

[e3] FindBlockNum(...): seems broken TxNum value #12009

wmitsuda opened this issue Sep 16, 2024 · 4 comments
Assignees
@JkLondon
Labels
erigon3 imp1 High importance
Milestone
3.0.0-alpha6

Comments

@wmitsuda
Copy link
Member

Using today's main, sepolia.

Node was working fine, then suddenly it entered an unrecoverable state. Restart doesn't work, it seems DB got inconsistent.

INFO[09-16|18:33:26.485] P2P                                      app=caplin peers=0
INFO[09-16|18:33:33.695] head updated                             head=0x62f81df94163466eb6024365e79010317db27e396e1253a6f7d5ce11749cc2f2 hash
=0x62f81df94163466eb6024365e79010317db27e396e1253a6f7d5ce11749cc2f2 number=6705172 execution=58.656917ms mgas/s=217.76 average mgas/s=215.79 c
ommit=189.25µs alloc=5.3GB sys=11.3GB
INFO[09-16|18:33:44.785] head updated                             head=0x76829ae6ed935537200f8cbabb98f5c18a0032ca2e732e06653075613b86dcf1 hash
=0x76829ae6ed935537200f8cbabb98f5c18a0032ca2e732e06653075613b86dcf1 number=6705173 execution=94.075459ms mgas/s=131.49 average mgas/s=194.72 c
ommit=1.967458ms alloc=4.0GB sys=11.3GB
INFO[09-16|18:33:59.302] head updated                             head=0xf6ee4b72b7c13012d94e4d181eae84045dcda197d8538a568b59374d8b15ee69 hash
=0xf6ee4b72b7c13012d94e4d181eae84045dcda197d8538a568b59374d8b15ee69 number=6705174 execution=41.8355ms mgas/s=228.76 average mgas/s=201.53 com
mit=182.666µs alloc=4.2GB sys=11.3GB
INFO[09-16|18:34:04.566] head updated                             head=0xa1c43d17f238bc8da86e26c3704544316cb721bde0e03a99db5ccd97e89b1607 hash
=0xa1c43d17f238bc8da86e26c3704544316cb721bde0e03a99db5ccd97e89b1607 number=6705175 execution=183.084166ms mgas/s=163.61 average mgas/s=195.21
commit=254.833µs alloc=4.6GB sys=11.3GB
INFO[09-16|18:34:20.320] head updated                             head=0x65f3d7a6481e1a9cb8cbc1b9e6172f24d72b8b688952ba5c67be13a5a0bd723d hash
=0x65f3d7a6481e1a9cb8cbc1b9e6172f24d72b8b688952ba5c67be13a5a0bd723d number=6705176 execution=47.73225ms mgas/s=177.08 average mgas/s=192.62 co
mmit=111.583µs alloc=5.0GB sys=11.3GB
INFO[09-16|18:34:26.486] P2P                                      app=caplin peers=0
INFO[09-16|18:34:57.063] Adding light client update               period=720
INFO[09-16|18:34:57.488] [4/6 Execution] Done Commit every block  blk=6705179 blks=3 blk/s=18.1 txs=305 tx/s=1.83k gas/s=258.55M buf=5.3MB/512
.0MB stepsInDB=0.00 step=198.0 alloc=4.5GB sys=11.3GB
INFO[09-16|18:34:57.746] head updated                             head=0x2ef5003ec9ef3da04805f0767c87ec7f91430499c21577f4758387efaed295d0 hash
=0x2ef5003ec9ef3da04805f0767c87ec7f91430499c21577f4758387efaed295d0 commit=139.25µs alloc=4.5GB sys=11.3GB
INFO[09-16|18:34:57.747] cl reorg                                 new_head_slot=5899074 fork_slot=5899071 old_canonical=0xa28fcfec529b147bfbc2
21011b1b44b6f836998de10aa4d9728f697187f2ecaf new_canonical=0x6c2e9f4384c66360f6ef1093b84834b44cbd8ed1b1518d4f300422dbcd8c64cd
INFO[09-16|18:35:26.487] P2P                                      app=caplin peers=1
INFO[09-16|18:35:34.059] [4/6 Execution] Done Commit every block  blk=6705181 blks=2 blk/s=19.1 txs=228 tx/s=2.18k gas/s=202.66M buf=4.2MB/512
.0MB stepsInDB=0.00 step=198.0 alloc=6.8GB sys=11.3GB
INFO[09-16|18:35:34.296] head updated                             head=0x281205fdd092035bf2306e03ad71b43bcbfc2012206c4081fcf7809e2790097c hash
=0x281205fdd092035bf2306e03ad71b43bcbfc2012206c4081fcf7809e2790097c commit=125.625µs alloc=6.8GB sys=11.3GB
INFO[09-16|18:35:34.296] cl reorg                                 new_head_slot=5899077 fork_slot=5899074 old_canonical=0x6c2e9f4384c66360f6ef
1093b84834b44cbd8ed1b1518d4f300422dbcd8c64cd new_canonical=0x78dff54a3409a83e703c86c796505caf585a9f1cbd3067db408e1acd22129679
INFO[09-16|18:36:24.447] [p2p] GoodPeers                          eth68=1
INFO[09-16|18:36:24.447] [mem] memory stats                       alloc=3.2GB sys=11.3GB
INFO[09-16|18:36:24.463] [txpool] stat                            pending=974 baseFee=554 queued=906 alloc=3.2GB sys=11.3GB
INFO[09-16|18:36:26.485] P2P                                      app=caplin peers=0
INFO[09-16|18:37:24.118] [4/6 Execution] Done Commit every block  blk=6705189 blks=8 blk/s=15.1 txs=1184 tx/s=2.22k gas/s=259.13M buf=16.6MB/5
12.0MB stepsInDB=0.00 step=198.0 alloc=3.1GB sys=11.3GB
INFO[09-16|18:37:24.445] cl reorg                                 new_head_slot=5899086 fork_slot=5899077 old_canonical=0x78dff54a3409a83e703c
86c796505caf585a9f1cbd3067db408e1acd22129679 new_canonical=0x14a92944edea64ba437ad3c7dcce7e767e485e1178b50ff903b0468f00960236
INFO[09-16|18:37:26.485] P2P                                      app=caplin peers=1
INFO[09-16|18:37:27.154] head updated                             head=0xc57e56b1b702df51beb61d7e94af203c2e1dbb0347d7102c42fd89e1e47cdf02 hash
=0xc57e56b1b702df51beb61d7e94af203c2e1dbb0347d7102c42fd89e1e47cdf02 commit=81.751042ms alloc=3.1GB sys=11.3GB
INFO[09-16|18:37:38.649] head updated                             head=0x58b4dc5296093cd42bd2be48417d88b66c6b9c73e84af15c07ffd550f248ded3 hash
=0x58b4dc5296093cd42bd2be48417d88b66c6b9c73e84af15c07ffd550f248ded3 number=6705190 execution=71.543792ms mgas/s=239.96 average mgas/s=198.54 c
ommit=91.25µs alloc=3.3GB sys=11.3GB
INFO[09-16|18:38:26.485] P2P                                      app=caplin peers=1
INFO[09-16|18:38:37.284] [snapshots:merge] Stat                   blocks=6.70M indices=6.70M alloc=2.4GB sys=11.3GB
INFO[09-16|18:38:39.749] [snapshots:blocks:retire] Stat           blocks=6.60M indices=6.60M alloc=2.3GB sys=11.3GB
WARN[09-16|18:38:44.548] Cannot update chain head                 hash=0xfbef5597530ee40b275df28b359e4441a37c9e71ae37ec7ead2c255cb15b22f0 err=
"updateForkChoice: [4/6 Execution] FindBlockNum(309449193): seems broken TxNum value: 0000000000000000 -> (64b6eb, 0); last in db: (6705194, 3
09449660)"
WARN[09-16|18:38:44.549] error executing clstage                  app=caplin stage=ForkChoice err="failed to compute and notify services of ne
w fork choice: failed to run forkchoice: execution Client RPC failed to retrieve ForkChoiceUpdate response, err: updateForkChoice: [4/6 Execut
ion] FindBlockNum(309449193): seems broken TxNum value: 0000000000000000 -> (64b6eb, 0); last in db: (6705194, 309449660)"
WARN[09-16|18:39:01.427] Cannot update chain head                 hash=0xb9ba16189c8d505f534fec4a671c4932560f09fd637eea326914680a3f8ba306 err=
"updateForkChoice: [4/6 Execution] FindBlockNum(309449193): seems broken TxNum value: 0000000000000000 -> (64b6ec, 0); last in db: (6705195, 3
09449979)"
WARN[09-16|18:39:01.427] error executing clstage                  app=caplin stage=ForkChoice err="failed to compute and notify services of ne
w fork choice: failed to run forkchoice: execution Client RPC failed to retrieve ForkChoiceUpdate response, err: updateForkChoice: [4/6 Execut
ion] FindBlockNum(309449193): seems broken TxNum value: 0000000000000000 -> (64b6ec, 0); last in db: (6705195, 309449979)"
WARN[09-16|18:39:19.023] Cannot update chain head                 hash=0xabcf83c7f5b5975203c0da0517d6e1b67e36a29dc988e359c24b619edea3ed59 err=
"updateForkChoice: [4/6 Execution] FindBlockNum(309449193): seems broken TxNum value: 0000000000000000 -> (64b6ed, 0); last in db: (6705196, 3
09450048)"
WARN[09-16|18:39:19.024] error executing clstage                  app=caplin stage=ForkChoice err="failed to compute and notify services of ne
w fork choice: failed to run forkchoice: execution Client RPC failed to retrieve ForkChoiceUpdate response, err: updateForkChoice: [4/6 Execut
ion] FindBlockNum(309449193): seems broken TxNum value: 0000000000000000 -> (64b6ed, 0); last in db: (6705196, 309450048)"
INFO[09-16|18:39:24.448] [p2p] GoodPeers                          eth68=1
INFO[09-16|18:39:24.448] [mem] memory stats                       alloc=2.8GB sys=11.3GB
INFO[09-16|18:39:24.462] [txpool] stat                            pending=660 baseFee=1277 queued=898 alloc=2.8GB sys=11.3GB
INFO[09-16|18:39:26.486] P2P                                      app=caplin peers=2
WARN[09-16|18:39:32.763] Cannot update chain head                 hash=0x494d34f29a44c80a3dcb1b23f6a75638e6f1ec7e3dbb8604b3fc48f0ee0dad7a err=
"updateForkChoice: [4/6 Execution] FindBlockNum(309449193): seems broken TxNum value: 0000000000000000 -> (64b6ee, 0); last in db: (6705197, 3
09450244)"
WARN[09-16|18:39:32.763] error executing clstage                  app=caplin stage=ForkChoice err="failed to compute and notify services of ne
w fork choice: failed to run forkchoice: execution Client RPC failed to retrieve ForkChoiceUpdate response, err: updateForkChoice: [4/6 Execut
ion] FindBlockNum(309449193): seems broken TxNum value: 0000000000000000 -> (64b6ee, 0); last in db: (6705197, 309450244)"
WARN[09-16|18:39:37.985] Cannot update chain head                 hash=0x3eeeca948dae8c83f24cb4acb7b06c10876bce57bfac5e259b9e81167a36ebab err=
"updateForkChoice: [4/6 Execution] FindBlockNum(309449193): seems broken TxNum value: 0000000000000000 -> (64b6ef, 0); last in db: (6705198, 3
09450304)"
WARN[09-16|18:39:37.985] error executing clstage                  app=caplin stage=ForkChoice err="failed to compute and notify services of ne
w fork choice: failed to run forkchoice: execution Client RPC failed to retrieve ForkChoiceUpdate response, err: updateForkChoice: [4/6 Execut
ion] FindBlockNum(309449193): seems broken TxNum value: 0000000000000000 -> (64b6ef, 0); last in db: (6705198, 309450304)"
WARN[09-16|18:39:56.893] Cannot update chain head                 hash=0x75a9e77912aabf4633e4df22cec110189b16f1248fa14f0dc00ad10688784b6c err=
"updateForkChoice: [4/6 Execution] FindBlockNum(309449193): seems broken TxNum value: 0000000000000000 -> (64b6f0, 0); last in db: (6705199, 3
09450403)"
WARN[09-16|18:39:56.893] error executing clstage                  app=caplin stage=ForkChoice err="failed to compute and notify services of ne
w fork choice: failed to run forkchoice: execution Client RPC failed to retrieve ForkChoiceUpdate response, err: updateForkChoice: [4/6 Execut
ion] FindBlockNum(309449193): seems broken TxNum value: 0000000000000000 -> (64b6f0, 0); last in db: (6705199, 309450403)"
WARN[09-16|18:40:04.047] Cannot update chain head                 hash=0x699b2a4be35670cd7788a652e8c3fdc2477b7eb83f551f34c362b537d2ddbc9a err=
"updateForkChoice: [4/6 Execution] FindBlockNum(309449193): seems broken TxNum value: 0000000000000000 -> (64b6f1, 0); last in db: (6705200, 3
09450830)"
WARN[09-16|18:40:04.047] error executing clstage                  app=caplin stage=ForkChoice err="failed to compute and notify services of ne
w fork choice: failed to run forkchoice: execution Client RPC failed to retrieve ForkChoiceUpdate response, err: updateForkChoice: [4/6 Execut
ion] FindBlockNum(309449193): seems broken TxNum value: 0000000000000000 -> (64b6f1, 0); last in db: (6705200, 309450830)"
WARN[09-16|18:40:16.405] Cannot update chain head                 hash=0xbecc0d44eb415a8f4413e77e31fbf1f7177410494c3e533827ac21a975eaecf6 err=
"updateForkChoice: [4/6 Execution] FindBlockNum(309449193): seems broken TxNum value: 0000000000000000 -> (64b6f2, 0); last in db: (6705201, 3
09450960)"
WARN[09-16|18:40:16.405] error executing clstage                  app=caplin stage=ForkChoice err="failed to compute and notify services of ne
w fork choice: failed to run forkchoice: execution Client RPC failed to retrieve ForkChoiceUpdate response, err: updateForkChoice: [4/6 Execut
ion] FindBlockNum(309449193): seems broken TxNum value: 0000000000000000 -> (64b6f2, 0); last in db: (6705201, 309450960)"
WARN[09-16|18:40:24.946] Cannot update chain head                 hash=0x15a2a648e99925d5a401a5a50875dce78889464e9d20e3336bc313801a2a282b err=
"updateForkChoice: [4/6 Execution] FindBlockNum(309449193): seems broken TxNum value: 0000000000000000 -> (64b6f3, 0); last in db: (6705202, 3
09451034)"
WARN[09-16|18:40:24.946] error executing clstage                  app=caplin stage=ForkChoice err="failed to compute and notify services of ne
w fork choice: failed to run forkchoice: execution Client RPC failed to retrieve ForkChoiceUpdate response, err: updateForkChoice: [4/6 Execut
ion] FindBlockNum(309449193): seems broken TxNum value: 0000000000000000 -> (64b6f3, 0); last in db: (6705202, 309451034)"
INFO[09-16|18:40:26.486] P2P                                      app=caplin peers=4
WARN[09-16|18:40:44.725] Cannot update chain head                 hash=0xde67d7dcd065dfc0d5a51efddafacfc6e39137030cf50bea19e62e302c2678d3 err=
"updateForkChoice: [4/6 Execution] FindBlockNum(309449193): seems broken TxNum value: 0000000000000000 -> (64b6f4, 0); last in db: (6705203, 3
09451609)"
WARN[09-16|18:40:44.725] error executing clstage                  app=caplin stage=ForkChoice err="failed to compute and notify services of ne
w fork choice: failed to run forkchoice: execution Client RPC failed to retrieve ForkChoiceUpdate response, err: updateForkChoice: [4/6 Execut
ion] FindBlockNum(309449193): seems broken TxNum value: 0000000000000000 -> (64b6f4, 0); last in db: (6705203, 309451609)"
WARN[09-16|18:40:48.600] Cannot update chain head                 hash=0x57923f615a5cf427ced8a96d8b08be0e4306ffcf102fd83c112e9dd4ef76f99d err=
"updateForkChoice: [4/6 Execution] FindBlockNum(309449193): seems broken TxNum value: 0000000000000000 -> (64b6f5, 0); last in db: (6705204, 3
09451696)"
WARN[09-16|18:40:48.600] error executing clstage                  app=caplin stage=ForkChoice err="failed to compute and notify services of ne
w fork choice: failed to run forkchoice: execution Client RPC failed to retrieve ForkChoiceUpdate response, err: updateForkChoice: [4/6 Execut
ion] FindBlockNum(309449193): seems broken TxNum value: 0000000000000000 -> (64b6f5, 0); last in db: (6705204, 309451696)"
WARN[09-16|18:41:08.657] Cannot update chain head                 hash=0xe09e674ded60d1838526d5b7e6f950eb60bcc1847c501b774911e760cce98f4a err=
"updateForkChoice: [4/6 Execution] FindBlockNum(309449193): seems broken TxNum value: 0000000000000000 -> (64b6f6, 0); last in db: (6705205, 3
09451767)"
@wmitsuda
Copy link
Member Author

using the "delete chaindata" workaround... FYI only 1 of my sepolia nodes went crazy, the other one is fine

@AskAlexSharov AskAlexSharov added imp1 High importance erigon3 labels Sep 17, 2024
@AskAlexSharov AskAlexSharov added this to the 3.0.0-alpha4 milestone Sep 17, 2024
@taratorio
Copy link
Member

Same here, happens on Amoy too

@AskAlexSharov
Copy link
Collaborator

probably we need add runtime-assert to catch earlier: #11211

@VBulikov VBulikov modified the milestones: 3.0.0-alpha4, 3.0.0-alpha5 Sep 27, 2024
@taratorio
Copy link
Member

happened on bor mainnet too, seems like restart (without deleting chaindata) helped

[DBUG] [10-04|15:13:26.147] Error while executing stage              err="[4/6 Execution] FindBlockNum(4607060128): seems broken TxNum value: 0000000000000000 -> (58712842, 0); db has: (0-62627030, 1-4607060209)"
[EROR] [10-04|15:13:26.147] Staged Sync                              err="[4/6 Execution] FindBlockNum(4607060128): seems broken TxNum value: 0000000000000000 -> (58712842, 0); db has: (0-62627030, 1-4607060209)"

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@JkLondon JkLondon

Labels
erigon3 imp1 High importance
Projects
None yet
Milestone
3.0.0-alpha6
Development

No branches or pull requests
5 participants
@wmitsuda @VBulikov @AskAlexSharov @JkLondon @taratorio