Marking block as invalid because of temporary world state root mismatch

[daniellehrner]

Description

I happened already two time that my node has a temporary World State Root mismatch. Temporary, because once the CL has recovered after Besu has sent an erroneous invalid block message, the same block can be imported without any changes in the running Besu instance.

The config file:

data-path="/var/lib/besu"
engine-jwt-secret="/var/lib/jwtsecret/jwt.hex"
rpc-http-enabled=true
metrics-enabled=true
logging="INFO"
rpc-http-api=["ETH","NET","WEB3", "DEBUG"]
Xsnapsync-server-enabled=true
Xsnapsync-synchronizer-flat-db-healing-enabled=true
Xbonsai-trie-log-pruning-enabled=true
Xbonsai-code-using-code-hash-enabled=true
Xbonsai-parallel-tx-processing-enabled=true

The error message:

2024-10-30 12:52:25.114+00:00 | vert.x-worker-thread-0 | ERROR | AbstractBlockProcessor | failed persisting block
java.lang.RuntimeException: World State Root does not match expected value, header 0x0787bd3cdf97a5ac4bf602e2404a07adee62838abad8a8ba22e3d4d963944b54 calculated 0x7483bd3302935cd3cd9e10d1db634fc0a04174d9aea4c8c1e0b00e515ea611a3
        at org.hyperledger.besu.ethereum.trie.diffbased.common.worldview.DiffBasedWorldState.verifyWorldStateRoot(DiffBasedWorldState.java:226)
        at org.hyperledger.besu.ethereum.trie.diffbased.common.worldview.DiffBasedWorldState.persist(DiffBasedWorldState.java:184)
        at org.hyperledger.besu.ethereum.mainnet.AbstractBlockProcessor.processBlock(AbstractBlockProcessor.java:233)
        at org.hyperledger.besu.ethereum.mainnet.BlockProcessor.processBlock(BlockProcessor.java:78)
        at org.hyperledger.besu.ethereum.MainnetBlockValidator.processBlock(MainnetBlockValidator.java:247)
        at org.hyperledger.besu.ethereum.MainnetBlockValidator.validateAndProcessBlock(MainnetBlockValidator.java:164)
        at org.hyperledger.besu.ethereum.MainnetBlockValidator.validateAndProcessBlock(MainnetBlockValidator.java:108)
        at org.hyperledger.besu.consensus.merge.blockcreation.MergeCoordinator.validateBlock(MergeCoordinator.java:557)
        at org.hyperledger.besu.consensus.merge.blockcreation.MergeCoordinator.rememberBlock(MergeCoordinator.java:587)
        at org.hyperledger.besu.consensus.merge.blockcreation.TransitionCoordinator.rememberBlock(TransitionCoordinator.java:157)
        at org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.engine.AbstractEngineNewPayload.syncResponse(AbstractEngineNewPayload.java:374)
        at org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.ExecutionEngineJsonRpcMethod.lambda$response$0(ExecutionEngineJsonRpcMethod.java:90)
        at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$1(ContextImpl.java:191)
        at io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:279)
        at io.vertx.core.impl.ContextImpl.lambda$internalExecuteBlocking$2(ContextImpl.java:210)
        at io.vertx.core.impl.TaskQueue.run(TaskQueue.java:76)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base/java.lang.Thread.run(Thread.java:1583)
2024-10-30 12:52:25.150+00:00 | vert.x-worker-thread-0 | INFO  | MainnetBlockValidator | Failed to process block 21078577 (0x22572bcd031b872c2f84ade258fa15dfd82deb3ad60e52bb5e83b02c35882974): Optional[World State Root does not match expected value, header 0x0787bd3cdf97a5ac4bf602e2404a07adee62838abad8a8ba22e3d4d963944b54 calculated 0x7483bd3302935cd3cd9e10d1db634fc0a04174d9aea4c8c1e0b00e515ea611a3], caused by java.lang.RuntimeException: World State Root does not match expected value, header 0x0787bd3cdf97a5ac4bf602e2404a07adee62838abad8a8ba22e3d4d963944b54 calculated 0x7483bd3302935cd3cd9e10d1db634fc0a04174d9aea4c8c1e0b00e515ea611a3
2024-10-30 12:52:25.153+00:00 | vert.x-worker-thread-0 | WARN  | AbstractEngineNewPayload | Invalid new payload: number: 21078577, hash: 0x22572bcd031b872c2f84ade258fa15dfd82deb3ad60e52bb5e83b02c35882974, parentHash: 0x6a24b47ea1288b302368281eb3134b19455e57a4083724561355e85275304689, latestValidHash: 0x6a24b47ea1288b302368281eb3134b19455e57a4083724561355e85275304689, status: INVALID, validationError: World State Root does not match expected value, header 0x0787bd3cdf97a5ac4bf602e2404a07adee62838abad8a8ba22e3d4d963944b54 calculated 0x7483bd3302935cd3cd9e10d1db634fc0a04174d9aea4c8c1e0b00e515ea611a3
2024-10-30 12:53:27.072+00:00 | vert.x-worker-thread-0 | INFO  | AbstractEngineForkchoiceUpdated | VALID for fork-choice-update: head: 0x6a24b47ea1288b302368281eb3134b19455e57a4083724561355e85275304689, finalized: 0x284dc25665922634656360afbef4fc1bc4123844a37caa4d07cdc6eb2f3d4de9, safeBlockHash: 0x824fe222e9f33c744f0777443fd8d307a1884e841dc52386a89548ee6aa652f3

Frequency: Has happened 2 times in as many weeks.