Stage 5 Verification failed on Beta network

[SurfingNerd]

We have another Stage 5 Verification failures detected:
Here is an example for a pending validator:

2024-12-11 13:07:31 Worker Client1 DEBUG consensus Block creation: Batch received for epoch 5168, total 20 contributions, with 1 unique transactions.
2024-12-11 13:07:32 Worker Client3 INFO import Imported #5168 0x5d34…f40b (1 txs, 62.97 Mgas, 80 ms, 96.92 KiB)
2024-12-11 13:07:32 Verifier #0 DEBUG engine Initializing SyncKeyGen with block_id: Latest
2024-12-11 13:07:33 Verifier #0 WARN client Stage 5 verification failed for #5168 (0x5d34…f40b)
Block is ancient (current best block: #5168).
2024-12-11 13:07:38 Worker Client2 INFO import 54/50 peers 15 MiB chain 0 bytes queue RPC: 0 conn, 0 req/s, 0 µs
2024-12-11 13:07:41 Worker Hbbft0 DEBUG engine do_validator_engine_actions.
2024-12-11 13:07:41 Worker Hbbft0 DEBUG engine early-epoch-end: handle_early_epoch_end.
2024-12-11 13:07:41 Worker Hbbft0 ERROR engine early-epoch-end: notify_about_validator_reconnect Could not find reconnected validator in flagged validators.
2024-12-11 13:07:41 Worker Hbbft0 ERROR engine early-epoch-end: notify_about_validator_reconnect Could not find reconnected validator in flagged validators.
2024-12-11 13:07:41 Worker Hbbft0 ERROR engine early-epoch-end: notify_about_validator_reconnect Could not find reconnected validator in flagged validators.
2024-12-11 13:07:41 Worker Hbbft0 ERROR engine early-epoch-end: notify_about_validator_reconnect Could not find reconnected validator in flagged validators.
2024-12-11 13:07:41 Worker Hbbft0 ERROR engine early-epoch-end: notify_about_validator_reconnect Could not find reconnected validator in flagged validators.
2024-12-11 13:08:00 Worker Client3 DEBUG consensus Block creation: Batch received for epoch 5169, total 76 contributions, with 4 unique transactions.
2024-12-11 13:08:00 Worker Client3 DEBUG engine Initializing SyncKeyGen with block_id: Latest
2024-12-11 13:08:02 Worker Hbbft1 DEBUG consensus Detected an attempt to send a hbbft contribution for block 5170 before the previous block was imported to the chain.
2024-12-11 13:08:02 Worker Client0 DEBUG engine Initializing SyncKeyGen with block_id: Number(5169)
2024-12-11 13:08:03 Worker Client0 INFO engine public master key: PublicKey(0b9c..7221)
2024-12-11 13:08:03 Worker Client0 INFO engine We are not part of the HoneyBadger validator set - running as regular node.
2024-12-11 13:08:04 Worker Client0 INFO engine removed 24 peers from reserved peers management.
2024-12-11 13:08:04 Worker Client0 INFO consensus POSDAO epoch changed from 29 to 30.
2024-12-11 13:08:04 Verifier #2 DEBUG engine Initializing SyncKeyGen with block_id: Number(5137)
2024-12-11 13:08:04 Worker Client0 INFO import Imported #5169 0xb49b…d16d (4 txs, 0.66 Mgas, 1904 ms, 1.38 KiB)
2024-12-11 13:08:05 Verifier #2 WARN client Stage 5 verification failed for #5169 (0xb49b…d16d)
Block is ancient (current best block: #5169).
2024-12-11 13:08:11 Worker Client3 INFO import 54/50 peers 15 MiB chain 0 bytes queue RPC: 0 conn, 0 req/s, 0 µs
2024-12-11 13:08:42 Worker Client3 INFO import 53/50 peers 15 MiB chain 0 bytes queue RPC: 0 conn, 0 req/s, 0 µs

[cryptonit]

to me it looks like it happens exact the block new epoch starts and just for validators which where part of the active set before and are not part of the new set

also it not as in the past lead to a crash or hand of the node it just recover by itself

if its that way its not even a major issue as it only affect non part of active set validators and just once

[SurfingNerd]

After setting up the Monitoring, we could not find any cases were a stage 5 verification has lead to a stalled node.
But we can analyse a case where a stage 5 verification did happen, but did not lead to a stalled network.
Hbbft13:

2025-01-10 09:54:04  Worker Client1 DEBUG consensus  Block creation: Batch received for epoch 35141, total 20 contributions, with 1 unique transactions.
2025-01-10 09:54:04  Worker Hbbft3 DEBUG consensus  Detected an attempt to send a hbbft contribution for block 35142 before the previous block was imported to the chain.
2025-01-10 09:54:04  Worker Hbbft3 DEBUG consensus  Detected an attempt to send a hbbft contribution for block 35142 before the previous block was imported to the chain.
2025-01-10 09:54:05  Worker Client3 INFO import  Imported #35141 0x4cb3…91f2 (1 txs, 62.94 Mgas, 634 ms, 96.92 KiB)
2025-01-10 09:54:05  Verifier #2 DEBUG engine  Initializing SyncKeyGen with block_id: Latest
2025-01-10 09:54:07  Verifier #2 WARN client  Stage 5 verification failed for #35141 (0x4cb3…91f2)
2025-01-10 09:54:07  Worker Hbbft1 INFO import  Imported #35141 0x4cb3…91f2 (1 txs, 62.94 Mgas, 1814 ms, 96.92 KiB)
2025-01-10 09:54:09  Worker Client2 DEBUG consensus  Block creation: Batch received for epoch 35142, total 0 contributions, with 0 unique transactions.
2025-01-10 09:54:09  Worker Client2 DEBUG engine  Initializing SyncKeyGen with block_id: Latest
2025-01-10 09:54:11  Worker Hbbft3 DEBUG consensus  Detected an attempt to send a hbbft contribution for block 35143 before the previous block was imported to the chain.
2025-01-10 09:54:11  Verifier #3 DEBUG engine  Initializing SyncKeyGen with block_id: Latest
2025-01-10 09:54:13  Verifier #3 DEBUG engine  Initializing SyncKeyGen with block_id: Number(35142)
2025-01-10 09:54:14  Verifier #3 INFO engine  public master key: PublicKey(004d..7b5b)
2025-01-10 09:54:14  Verifier #3 INFO engine  We are not part of the HoneyBadger validator set - running as regular node.
2025-01-10 09:54:14  Verifier #3 INFO engine  removed 39 peers from reserved peers management.
2025-01-10 09:54:14  Verifier #3 INFO consensus  POSDAO epoch changed from 127 to 128.
2025-01-10 09:54:15  Worker Hbbft2 INFO import  Imported #35142 0x4165…20b3 (0 txs, 0.00 Mgas, 2915 ms, 0.58 KiB)
2025-01-10 09:54:20  Worker Client1 INFO import   66/148 peers   22 MiB chain 0 bytes queue  RPC:  0 conn,    0 req/s,    0 µs
2025-01-10 09:54:50  Worker Client0 INFO import   68/148 peers   22 MiB chain 0 bytes queue  RPC:  0 conn,    0 req/s,    0 µs

[SurfingNerd]

We encountered a Stage 3 Verification Error: #68
Nodes have been experimental updated with a fix for that, that maybe can fix Stage 5 verification errors as well.
Stage 4 verification Errors are flagged as Stage 5 verification errors in the logs.

[SurfingNerd]

For the Time Range we have been Analytics, i found no node, that got stuck because of a Stage 5 verification error.
Maybe we need to work on a testnetwork, to make the problem reproducable.