[Bug]: Consensus failure lottery

[fmorency]

Is there an existing issue for this?

• I have searched the existing issues

What happened?

We added CosmWasm support to our chain and tried to upgrade our devnet. However, we stumbled on

panic: error loading last version: failed to load latest version: version of store wasm mismatch root store's version; expected 571999 got 0; new stores should be added using StoreUpgrades

We wrote interchaintest/chain_upgrade_test.go in liftedinit/manifest-ledger#118 and sometime it fails with the error above, sometime everything is fine and the upgrade/test is successful.

I'm not sure what's going on.

The upgrade handler can be found in app/upgrades/next/upgrades.go

Cosmos SDK Version

0.50.11

How to reproduce?

Checkout the branch related to liftedinit/manifest-ledger#118

Run make local-image and make ictest-chain-upgrade.

The test might pass. Run it again until it fails.

[fmorency]

@Reecepbcups wrote a workaround in Reecepbcups@72e651e

It seems it was resolved in v50 but it recently came back up on many networks around the same time.

[Reecepbcups]

also ref from the hub: cosmos/gaia#2313 (uses my above patch as a work around)

[fmorency]

Update

Upon reaching the upgrade height, nodes automatically shut down (and restart) to apply the new upgrade. In smaller blockchain networks, this simultaneous shutdown of multiple nodes can lead to a loss of consensus if a sufficient number of nodes become temporarily inactive. The Cosmos SDK does not wait for a majority of nodes to reach the upgrade height before initiating the shutdown process. Consequently, if consensus is lost during this period, all nodes may shut down, resulting in the observed error.

To maintain consensus during upgrades, we should monitor which nodes have been updated and which have not.

After bumping the number of validators from 2 to 16 in our chain-upgrade test, the error did not reoccur. However, without implementing a proper fix, the issue may still occur due to (lack of) chance.

[fmorency]

Update

The chain-upgrade test consistently passes with only 4 nodes. However, the same update fails on our devnet. The main difference between the test and our real devnet is Cosmovisor.

I'm starting to suspect Cosmovisor.

Here's the interesting bit

No Cosmovisor: The application exits itself

7:02PM INF caught signal module=server signal=terminated

Cosmovisor: The application receives an interrupt from Cosmovisor

cosmovisor[3613069]: 2:52PM INF daemon shutting down in an attempt to restart module=cosmovisor
cosmovisor[3613069]: 2:52PM INF sent interrupt to app, waiting for exit module=cosmovisor
cosmovisor[3613092]: 2:52PM INF caught signal module=server signal=interrupt