fix(->abci-patch): abci removing logic

[Reecepbcups]

Using this to test ci for all the cases

[Reecepbcups]

on val removal it is not clearing out properly (re-add beginblocker for unbonding case)

        ===== TEST REMOVE VALIDATOR =====
    logger.go:130: 2024-04-06T13:22:06.229-0500 INFO    About to start container        {"chain_id": "poa-1", "test": "TestPOABase", "image": "poa:local", "test_name": "TestPOABase", "command": "poad tx poa set-power cosmosvaloper1mjjqmnmt0dq9yzu3ryn77yts7w5hnqx5kcrjre 9000000000000 --node http://poa-1-val-0-TestPOABase:26657 --home /var/cosmos-chain/poa --chain-id poa-1 --from acc0-poa-1-bnt --keyring-backend test --output=json --yes --gas 500000 --unsafe", "hostname": "TestPOABase-vnmthk", "container": "TestPOABase-vnmthk"}

ENDBLOCK POA valUpdates before:
 - ed25519:"\227DC^\205\254\221Z\tH\301\245\021+X\251\303B\221W{Zx'=\323\213Q\007\225\267\355" : 1

isNewVal false
power 1
err <nil>
POA valUpdates after:
 - ed25519:"\227DC^\205\254\221Z\tH\301\245\021+X\251\303B\221W{Zx'=\323\213Q\007\225\267\355" : 1
6:22PM INF finalized block block_app_hash=23F602C03677D4FA08DF1AB5D406C16D7D589B0C96E415C61057D265A5C67B94 height=35 module=state num_txs_res=0 num_val_updates=1
6:22PM INF executed block app_hash=23F602C03677D4FA08DF1AB5D406C16D7D589B0C96E415C61057D265A5C67B94 height=35 module=state
6:22PM INF updates to validators module=state updates=8F5F44D55A2464100D79CBEC1D241A515F54DB54:1
6:22PM INF committed state block_app_hash=F17507D1B7DFEBB39B0CFAD6CF1CD3DBCBBDC736FBE9CDAB2A0F01CC5F933934 height=35 module=state
6:22PM INF indexed block events height=35 module=txindex
6:22PM INF Timed out dur=1995.751036 height=36 module=consensus round=0 step=RoundStepNewHeight
6:22PM INF received proposal module=consensus proposal="Proposal{36/0 (3F110BBD05BA4DAA5F0D14FDAB312EF9A0CFE6AA6414C49155F63D228B885BD9:1:B58C32C85316, -1) 2C942C70747C @ 2024-04-06T18:22:07.639881154Z}" proposer=8F5F44D55A2464100D79CBEC1D241A515F54DB54
6:22PM INF received complete proposal block hash=3F110BBD05BA4DAA5F0D14FDAB312EF9A0CFE6AA6414C49155F63D228B885BD9 height=36 module=consensus
6:22PM INF finalizing commit of block hash=3F110BBD05BA4DAA5F0D14FDAB312EF9A0CFE6AA6414C49155F63D228B885BD9 height=36 module=consensus num_txs=1 root=23F602C03677D4FA08DF1AB5D406C16D7D589B0C96E415C61057D265A5C67B94

ENDBLOCK POA valUpdates before:
 - ed25519:"\227DC^\205\254\221Z\tH\301\245\021+X\251\303B\221W{Zx'=\323\213Q\007\225\267\355" : 1

isNewVal false
power 9000000
err <nil>
POA valUpdates after:
 - ed25519:"\227DC^\205\254\221Z\tH\301\245\021+X\251\303B\221W{Zx'=\323\213Q\007\225\267\355" : 1
6:22PM INF finalized block block_app_hash=45B474F90A2AEE9376C03FD57F564D239FD5507A565AB6F01AE3D4F5F5DE2860 height=36 module=state num_txs_res=1 num_val_updates=2
6:22PM INF executed block app_hash=45B474F90A2AEE9376C03FD57F564D239FD5507A565AB6F01AE3D4F5F5DE2860 height=36 module=state
6:22PM INF updates to validators module=state updates=8F5F44D55A2464100D79CBEC1D241A515F54DB54:9000000,8F5F44D55A2464100D79CBEC1D241A515F54DB54:9000000
6:22PM ERR CONSENSUS FAILURE!!! err="failed to apply block; error commit failed for application: changing validator set: duplicate entry Validator{8F5F44D55A2464100D79CBEC1D241A515F54DB54 PubKeyEd25519{9744435E85AC915A0948C1A5112B58A9C34291577B5A78273DD38B510795B7ED} VP:9000000 A:0} in [Validator{8F5F44D55A2464100D79CBEC1D241A515F54DB54 PubKeyEd25519{9744435E85AC915A0948C1A5112B58A9C34291577B5A78273DD38B510795B7ED} VP:9000000 A:0} Validator{8F5F44D55A2464100D79CBEC1D241A515F54DB54 PubKeyEd25519{9744435E85AC915A0948C1A5112B58A9C34291577B5A78273DD38B510795B7ED} VP:9000000 A:0}]" module=consensus stack="goroutine 467 [running]:\nruntime/debug.Stack()\n\truntime/debug/stack.go:24 +0x5e\ngithub.com/cometbft/cometbft/consensus.(*State).receiveRoutine.func2()\n\tgithub.com/cometbft/[email protected]/consensus/state.go:800 +0x46\npanic({0x273aba0?, 0xc0056af470?})\n\truntime/panic.go:914 +0x21f\ngithub.com/cometbft/cometbft/consensus.(*State).finalizeCommit(0xc001178380, 0x24)\n\tgithub.com/cometbft/[email protected]/consensus/state.go:1779 +0xda5\ngithub.com/cometbft/cometbft/consensus.(*State).tryFinalizeCommit(0xc001178380, 0x24)\n\tgithub.com/cometbft/[email protected]/consensus/state.go:1681 +0x2f6\ngithub.com/cometbft/cometbft/consensus.(*State).enterCommit.func1()\n\tgithub.com/cometbft/[email protected]/consensus/state.go:1616 +0x9c\ngithub.com/cometbft/cometbft/consensus.(*State).enterCommit(0xc001178380, 0x24, 0x0)\n\tgithub.com/cometbft/[email protected]/consensus/state.go:1654 +0xc5b\ngithub.com/cometbft/cometbft/consensus.(*State).addVote(0xc001178380, 0xc000bfcf70, {0x0, 0x0})\n\tgithub.com/cometbft/[email protected]/consensus/state.go:2324 +0x1d4d\ngithub.com/cometbft/cometbft/consensus.(*State).tryAddVote(0xc001178380, 0xc000bfcf70, {0x0?, 0x4be890?})\n\tgithub.com/cometbft/[email protected]/consensus/state.go:2056 +0x26\ngithub.com/cometbft/cometbft/consensus.(*State).handleMsg(0xc001178380, {{0x3a36d00, 0xc0013278c8}, {0x0, 0x0}})\n\tgithub.com/cometbft/[email protected]/consensus/state.go:928 +0x3ce\ngithub.com/cometbft/cometbft/consensus.(*State).receiveRoutine(0xc001178380, 0x0)\n\tgithub.com/cometbft/[email protected]/consensus/state.go:855 +0x4bc\ncreated by github.com/cometbft/cometbft/consensus.(*State).OnStart in goroutine 1\n\tgithub.com/cometbft/[email protected]/consensus/state.go:397 +0x10c\n"
6:22PM INF service stop impl=baseWAL module=consensus msg="Stopping baseWAL service" wal=/var/cosmos-chain/poa/data/cs.wal/wal
6:22PM INF service stop impl=Group module=consensus msg="Stopping Group service" wal=/var/cosmos-chain/poa/data/cs.wal/wal

[Reecepbcups]

val-add passes locally w/ fresh image & test cache clear, jail fails for empty set

11:42PM INF finalizing commit of block hash=8FC23C769BEB23DB973F8416F00CDAF3C14D0B703FEEBDDF57A09C5A820C4C40 height=7 module=consensus num_txs=0 root=A08936D4E1774099A4B410DD3F23A5E03E7EED6CD2DAF0BB86B472E43DBF7E30
11:42PM INF ValUpdate Before module=x/poa power=0 pubkey="ed25519:\"T\\266\\350^\\240\\315\\361e\\017\\347\\010cD2\\337M.\\002\\032\\024l\\211\\005\\272a\\253\\223q\\253\\227\\006r\" "
11:42PM INF ValUpdate After module=x/poa power=0 pubkey="ed25519:\"T\\266\\350^\\240\\315\\361e\\017\\347\\010cD2\\337M.\\002\\032\\024l\\211\\005\\272a\\253\\223q\\253\\227\\006r\" "
11:42PM INF finalized block block_app_hash=35B202CA0696E576DFF1CF57CD7FF7B18FF35CE237934DB2DF0E41231F3063F4 height=7 module=state num_txs_res=0 num_val_updates=4
11:42PM INF executed block app_hash=35B202CA0696E576DFF1CF57CD7FF7B18FF35CE237934DB2DF0E41231F3063F4 height=7 module=state
11:42PM INF updates to validators module=state updates=AF3D893E820B2FA506F23010BF7C7E9F35F01DF9:0,2D015BB7310A0FE01D52DC101B401E9F2BEBA75C:0,241F857144D96A9F7D7BACBF217D2E27D47D95A1:0,8F9A0009B36158AA6CA3DAF3D23198AE5A3EED5D:0
11:42PM ERR CONSENSUS FAILURE!!! err="failed to apply block; error commit failed for application: changing validator set: applying the validator changes would result in empty set"

[Reecepbcups]

BOOOYAAAAAAAA