We read every piece of feedback, and take your input very seriously.
To see all available qualifiers, see our documentation.
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Using today's main, sepolia.
Node was working fine, then suddenly it entered an unrecoverable state. Restart doesn't work, it seems DB got inconsistent.
INFO[09-16|18:33:26.485] P2P app=caplin peers=0 INFO[09-16|18:33:33.695] head updated head=0x62f81df94163466eb6024365e79010317db27e396e1253a6f7d5ce11749cc2f2 hash =0x62f81df94163466eb6024365e79010317db27e396e1253a6f7d5ce11749cc2f2 number=6705172 execution=58.656917ms mgas/s=217.76 average mgas/s=215.79 c ommit=189.25µs alloc=5.3GB sys=11.3GB INFO[09-16|18:33:44.785] head updated head=0x76829ae6ed935537200f8cbabb98f5c18a0032ca2e732e06653075613b86dcf1 hash =0x76829ae6ed935537200f8cbabb98f5c18a0032ca2e732e06653075613b86dcf1 number=6705173 execution=94.075459ms mgas/s=131.49 average mgas/s=194.72 c ommit=1.967458ms alloc=4.0GB sys=11.3GB INFO[09-16|18:33:59.302] head updated head=0xf6ee4b72b7c13012d94e4d181eae84045dcda197d8538a568b59374d8b15ee69 hash =0xf6ee4b72b7c13012d94e4d181eae84045dcda197d8538a568b59374d8b15ee69 number=6705174 execution=41.8355ms mgas/s=228.76 average mgas/s=201.53 com mit=182.666µs alloc=4.2GB sys=11.3GB INFO[09-16|18:34:04.566] head updated head=0xa1c43d17f238bc8da86e26c3704544316cb721bde0e03a99db5ccd97e89b1607 hash =0xa1c43d17f238bc8da86e26c3704544316cb721bde0e03a99db5ccd97e89b1607 number=6705175 execution=183.084166ms mgas/s=163.61 average mgas/s=195.21 commit=254.833µs alloc=4.6GB sys=11.3GB INFO[09-16|18:34:20.320] head updated head=0x65f3d7a6481e1a9cb8cbc1b9e6172f24d72b8b688952ba5c67be13a5a0bd723d hash =0x65f3d7a6481e1a9cb8cbc1b9e6172f24d72b8b688952ba5c67be13a5a0bd723d number=6705176 execution=47.73225ms mgas/s=177.08 average mgas/s=192.62 co mmit=111.583µs alloc=5.0GB sys=11.3GB INFO[09-16|18:34:26.486] P2P app=caplin peers=0 INFO[09-16|18:34:57.063] Adding light client update period=720 INFO[09-16|18:34:57.488] [4/6 Execution] Done Commit every block blk=6705179 blks=3 blk/s=18.1 txs=305 tx/s=1.83k gas/s=258.55M buf=5.3MB/512 .0MB stepsInDB=0.00 step=198.0 alloc=4.5GB sys=11.3GB INFO[09-16|18:34:57.746] head updated head=0x2ef5003ec9ef3da04805f0767c87ec7f91430499c21577f4758387efaed295d0 hash =0x2ef5003ec9ef3da04805f0767c87ec7f91430499c21577f4758387efaed295d0 commit=139.25µs alloc=4.5GB sys=11.3GB INFO[09-16|18:34:57.747] cl reorg new_head_slot=5899074 fork_slot=5899071 old_canonical=0xa28fcfec529b147bfbc2 21011b1b44b6f836998de10aa4d9728f697187f2ecaf new_canonical=0x6c2e9f4384c66360f6ef1093b84834b44cbd8ed1b1518d4f300422dbcd8c64cd INFO[09-16|18:35:26.487] P2P app=caplin peers=1 INFO[09-16|18:35:34.059] [4/6 Execution] Done Commit every block blk=6705181 blks=2 blk/s=19.1 txs=228 tx/s=2.18k gas/s=202.66M buf=4.2MB/512 .0MB stepsInDB=0.00 step=198.0 alloc=6.8GB sys=11.3GB INFO[09-16|18:35:34.296] head updated head=0x281205fdd092035bf2306e03ad71b43bcbfc2012206c4081fcf7809e2790097c hash =0x281205fdd092035bf2306e03ad71b43bcbfc2012206c4081fcf7809e2790097c commit=125.625µs alloc=6.8GB sys=11.3GB INFO[09-16|18:35:34.296] cl reorg new_head_slot=5899077 fork_slot=5899074 old_canonical=0x6c2e9f4384c66360f6ef 1093b84834b44cbd8ed1b1518d4f300422dbcd8c64cd new_canonical=0x78dff54a3409a83e703c86c796505caf585a9f1cbd3067db408e1acd22129679 INFO[09-16|18:36:24.447] [p2p] GoodPeers eth68=1 INFO[09-16|18:36:24.447] [mem] memory stats alloc=3.2GB sys=11.3GB INFO[09-16|18:36:24.463] [txpool] stat pending=974 baseFee=554 queued=906 alloc=3.2GB sys=11.3GB INFO[09-16|18:36:26.485] P2P app=caplin peers=0 INFO[09-16|18:37:24.118] [4/6 Execution] Done Commit every block blk=6705189 blks=8 blk/s=15.1 txs=1184 tx/s=2.22k gas/s=259.13M buf=16.6MB/5 12.0MB stepsInDB=0.00 step=198.0 alloc=3.1GB sys=11.3GB INFO[09-16|18:37:24.445] cl reorg new_head_slot=5899086 fork_slot=5899077 old_canonical=0x78dff54a3409a83e703c 86c796505caf585a9f1cbd3067db408e1acd22129679 new_canonical=0x14a92944edea64ba437ad3c7dcce7e767e485e1178b50ff903b0468f00960236 INFO[09-16|18:37:26.485] P2P app=caplin peers=1 INFO[09-16|18:37:27.154] head updated head=0xc57e56b1b702df51beb61d7e94af203c2e1dbb0347d7102c42fd89e1e47cdf02 hash =0xc57e56b1b702df51beb61d7e94af203c2e1dbb0347d7102c42fd89e1e47cdf02 commit=81.751042ms alloc=3.1GB sys=11.3GB INFO[09-16|18:37:38.649] head updated head=0x58b4dc5296093cd42bd2be48417d88b66c6b9c73e84af15c07ffd550f248ded3 hash =0x58b4dc5296093cd42bd2be48417d88b66c6b9c73e84af15c07ffd550f248ded3 number=6705190 execution=71.543792ms mgas/s=239.96 average mgas/s=198.54 c ommit=91.25µs alloc=3.3GB sys=11.3GB INFO[09-16|18:38:26.485] P2P app=caplin peers=1 INFO[09-16|18:38:37.284] [snapshots:merge] Stat blocks=6.70M indices=6.70M alloc=2.4GB sys=11.3GB INFO[09-16|18:38:39.749] [snapshots:blocks:retire] Stat blocks=6.60M indices=6.60M alloc=2.3GB sys=11.3GB WARN[09-16|18:38:44.548] Cannot update chain head hash=0xfbef5597530ee40b275df28b359e4441a37c9e71ae37ec7ead2c255cb15b22f0 err= "updateForkChoice: [4/6 Execution] FindBlockNum(309449193): seems broken TxNum value: 0000000000000000 -> (64b6eb, 0); last in db: (6705194, 3 09449660)" WARN[09-16|18:38:44.549] error executing clstage app=caplin stage=ForkChoice err="failed to compute and notify services of ne w fork choice: failed to run forkchoice: execution Client RPC failed to retrieve ForkChoiceUpdate response, err: updateForkChoice: [4/6 Execut ion] FindBlockNum(309449193): seems broken TxNum value: 0000000000000000 -> (64b6eb, 0); last in db: (6705194, 309449660)" WARN[09-16|18:39:01.427] Cannot update chain head hash=0xb9ba16189c8d505f534fec4a671c4932560f09fd637eea326914680a3f8ba306 err= "updateForkChoice: [4/6 Execution] FindBlockNum(309449193): seems broken TxNum value: 0000000000000000 -> (64b6ec, 0); last in db: (6705195, 3 09449979)" WARN[09-16|18:39:01.427] error executing clstage app=caplin stage=ForkChoice err="failed to compute and notify services of ne w fork choice: failed to run forkchoice: execution Client RPC failed to retrieve ForkChoiceUpdate response, err: updateForkChoice: [4/6 Execut ion] FindBlockNum(309449193): seems broken TxNum value: 0000000000000000 -> (64b6ec, 0); last in db: (6705195, 309449979)" WARN[09-16|18:39:19.023] Cannot update chain head hash=0xabcf83c7f5b5975203c0da0517d6e1b67e36a29dc988e359c24b619edea3ed59 err= "updateForkChoice: [4/6 Execution] FindBlockNum(309449193): seems broken TxNum value: 0000000000000000 -> (64b6ed, 0); last in db: (6705196, 3 09450048)" WARN[09-16|18:39:19.024] error executing clstage app=caplin stage=ForkChoice err="failed to compute and notify services of ne w fork choice: failed to run forkchoice: execution Client RPC failed to retrieve ForkChoiceUpdate response, err: updateForkChoice: [4/6 Execut ion] FindBlockNum(309449193): seems broken TxNum value: 0000000000000000 -> (64b6ed, 0); last in db: (6705196, 309450048)" INFO[09-16|18:39:24.448] [p2p] GoodPeers eth68=1 INFO[09-16|18:39:24.448] [mem] memory stats alloc=2.8GB sys=11.3GB INFO[09-16|18:39:24.462] [txpool] stat pending=660 baseFee=1277 queued=898 alloc=2.8GB sys=11.3GB INFO[09-16|18:39:26.486] P2P app=caplin peers=2 WARN[09-16|18:39:32.763] Cannot update chain head hash=0x494d34f29a44c80a3dcb1b23f6a75638e6f1ec7e3dbb8604b3fc48f0ee0dad7a err= "updateForkChoice: [4/6 Execution] FindBlockNum(309449193): seems broken TxNum value: 0000000000000000 -> (64b6ee, 0); last in db: (6705197, 3 09450244)" WARN[09-16|18:39:32.763] error executing clstage app=caplin stage=ForkChoice err="failed to compute and notify services of ne w fork choice: failed to run forkchoice: execution Client RPC failed to retrieve ForkChoiceUpdate response, err: updateForkChoice: [4/6 Execut ion] FindBlockNum(309449193): seems broken TxNum value: 0000000000000000 -> (64b6ee, 0); last in db: (6705197, 309450244)" WARN[09-16|18:39:37.985] Cannot update chain head hash=0x3eeeca948dae8c83f24cb4acb7b06c10876bce57bfac5e259b9e81167a36ebab err= "updateForkChoice: [4/6 Execution] FindBlockNum(309449193): seems broken TxNum value: 0000000000000000 -> (64b6ef, 0); last in db: (6705198, 3 09450304)" WARN[09-16|18:39:37.985] error executing clstage app=caplin stage=ForkChoice err="failed to compute and notify services of ne w fork choice: failed to run forkchoice: execution Client RPC failed to retrieve ForkChoiceUpdate response, err: updateForkChoice: [4/6 Execut ion] FindBlockNum(309449193): seems broken TxNum value: 0000000000000000 -> (64b6ef, 0); last in db: (6705198, 309450304)" WARN[09-16|18:39:56.893] Cannot update chain head hash=0x75a9e77912aabf4633e4df22cec110189b16f1248fa14f0dc00ad10688784b6c err= "updateForkChoice: [4/6 Execution] FindBlockNum(309449193): seems broken TxNum value: 0000000000000000 -> (64b6f0, 0); last in db: (6705199, 3 09450403)" WARN[09-16|18:39:56.893] error executing clstage app=caplin stage=ForkChoice err="failed to compute and notify services of ne w fork choice: failed to run forkchoice: execution Client RPC failed to retrieve ForkChoiceUpdate response, err: updateForkChoice: [4/6 Execut ion] FindBlockNum(309449193): seems broken TxNum value: 0000000000000000 -> (64b6f0, 0); last in db: (6705199, 309450403)" WARN[09-16|18:40:04.047] Cannot update chain head hash=0x699b2a4be35670cd7788a652e8c3fdc2477b7eb83f551f34c362b537d2ddbc9a err= "updateForkChoice: [4/6 Execution] FindBlockNum(309449193): seems broken TxNum value: 0000000000000000 -> (64b6f1, 0); last in db: (6705200, 3 09450830)" WARN[09-16|18:40:04.047] error executing clstage app=caplin stage=ForkChoice err="failed to compute and notify services of ne w fork choice: failed to run forkchoice: execution Client RPC failed to retrieve ForkChoiceUpdate response, err: updateForkChoice: [4/6 Execut ion] FindBlockNum(309449193): seems broken TxNum value: 0000000000000000 -> (64b6f1, 0); last in db: (6705200, 309450830)" WARN[09-16|18:40:16.405] Cannot update chain head hash=0xbecc0d44eb415a8f4413e77e31fbf1f7177410494c3e533827ac21a975eaecf6 err= "updateForkChoice: [4/6 Execution] FindBlockNum(309449193): seems broken TxNum value: 0000000000000000 -> (64b6f2, 0); last in db: (6705201, 3 09450960)" WARN[09-16|18:40:16.405] error executing clstage app=caplin stage=ForkChoice err="failed to compute and notify services of ne w fork choice: failed to run forkchoice: execution Client RPC failed to retrieve ForkChoiceUpdate response, err: updateForkChoice: [4/6 Execut ion] FindBlockNum(309449193): seems broken TxNum value: 0000000000000000 -> (64b6f2, 0); last in db: (6705201, 309450960)" WARN[09-16|18:40:24.946] Cannot update chain head hash=0x15a2a648e99925d5a401a5a50875dce78889464e9d20e3336bc313801a2a282b err= "updateForkChoice: [4/6 Execution] FindBlockNum(309449193): seems broken TxNum value: 0000000000000000 -> (64b6f3, 0); last in db: (6705202, 3 09451034)" WARN[09-16|18:40:24.946] error executing clstage app=caplin stage=ForkChoice err="failed to compute and notify services of ne w fork choice: failed to run forkchoice: execution Client RPC failed to retrieve ForkChoiceUpdate response, err: updateForkChoice: [4/6 Execut ion] FindBlockNum(309449193): seems broken TxNum value: 0000000000000000 -> (64b6f3, 0); last in db: (6705202, 309451034)" INFO[09-16|18:40:26.486] P2P app=caplin peers=4 WARN[09-16|18:40:44.725] Cannot update chain head hash=0xde67d7dcd065dfc0d5a51efddafacfc6e39137030cf50bea19e62e302c2678d3 err= "updateForkChoice: [4/6 Execution] FindBlockNum(309449193): seems broken TxNum value: 0000000000000000 -> (64b6f4, 0); last in db: (6705203, 3 09451609)" WARN[09-16|18:40:44.725] error executing clstage app=caplin stage=ForkChoice err="failed to compute and notify services of ne w fork choice: failed to run forkchoice: execution Client RPC failed to retrieve ForkChoiceUpdate response, err: updateForkChoice: [4/6 Execut ion] FindBlockNum(309449193): seems broken TxNum value: 0000000000000000 -> (64b6f4, 0); last in db: (6705203, 309451609)" WARN[09-16|18:40:48.600] Cannot update chain head hash=0x57923f615a5cf427ced8a96d8b08be0e4306ffcf102fd83c112e9dd4ef76f99d err= "updateForkChoice: [4/6 Execution] FindBlockNum(309449193): seems broken TxNum value: 0000000000000000 -> (64b6f5, 0); last in db: (6705204, 3 09451696)" WARN[09-16|18:40:48.600] error executing clstage app=caplin stage=ForkChoice err="failed to compute and notify services of ne w fork choice: failed to run forkchoice: execution Client RPC failed to retrieve ForkChoiceUpdate response, err: updateForkChoice: [4/6 Execut ion] FindBlockNum(309449193): seems broken TxNum value: 0000000000000000 -> (64b6f5, 0); last in db: (6705204, 309451696)" WARN[09-16|18:41:08.657] Cannot update chain head hash=0xe09e674ded60d1838526d5b7e6f950eb60bcc1847c501b774911e760cce98f4a err= "updateForkChoice: [4/6 Execution] FindBlockNum(309449193): seems broken TxNum value: 0000000000000000 -> (64b6f6, 0); last in db: (6705205, 3 09451767)"
The text was updated successfully, but these errors were encountered:
using the "delete chaindata" workaround... FYI only 1 of my sepolia nodes went crazy, the other one is fine
Sorry, something went wrong.
Same here, happens on Amoy too
probably we need add runtime-assert to catch earlier: #11211
happened on bor mainnet too, seems like restart (without deleting chaindata) helped
[DBUG] [10-04|15:13:26.147] Error while executing stage err="[4/6 Execution] FindBlockNum(4607060128): seems broken TxNum value: 0000000000000000 -> (58712842, 0); db has: (0-62627030, 1-4607060209)" [EROR] [10-04|15:13:26.147] Staged Sync err="[4/6 Execution] FindBlockNum(4607060128): seems broken TxNum value: 0000000000000000 -> (58712842, 0); db has: (0-62627030, 1-4607060209)"
JkLondon
No branches or pull requests
Using today's main, sepolia.
Node was working fine, then suddenly it entered an unrecoverable state. Restart doesn't work, it seems DB got inconsistent.
The text was updated successfully, but these errors were encountered: