[Consensus Observer] Add skeleton support for execution pool.

[JoshLind]

Description

This PR adds skeleton support to consensus observer (CO) for execution pool. Specifically, it makes the following modifications:

• Add a new ordered block with window message type (for execution pool blocks). The exact contents of this message are still TBD.
• Add basic flow control for handling the new message type, including stubs of methods to be implemented later.
• Adds new metrics for invalid messages, and cleans up some of the metric flow.

The PR will be squashed, but it has the following loose commits:

1. Basic skeleton code for the new message type and flow.
2. New and existing message metric refactors.
3. Simple skeleton code for window processing.

The new functionality is all gated by the dummy on-chain config value, and is off by default. Will continue building on this.

Testing Plan

Existing test infrastructure. New tests will be added to the new flows once the stubs are completed.

[trunk-io]

⏱️ 2h 11m total CI duration on this PR

Slowest 15 Jobs	Cumulative Duration	Recent Runs
execution-performance / single-node-performance	54m	🟥 🟩 🟥
test-target-determinator	18m	🟩 🟩 🟩
execution-performance / test-target-determinator	15m	🟩 🟩 🟩
check-dynamic-deps	10m	🟩 🟩 🟩 🟩
rust-doc-tests	6m	🟩
rust-doc-tests	6m	🟩
rust-doc-tests	6m	🟩
rust-cargo-deny	6m	🟩 🟩 🟥
fetch-last-released-docker-image-tag	5m	🟩 🟩 🟩
semgrep/ci	2m	🟩 🟩 🟩 🟩
general-lints	1m	🟩 🟩 🟩
file_change_determinator	34s	🟩 🟩 🟩
file_change_determinator	33s	⬜ 🟩 🟩 🟩
permission-check	10s	🟩 🟩 🟩 🟩
permission-check	7s	🟩 🟩 🟩

🚨 2 jobs on the last run were significantly faster/slower than expected

Job	Duration	vs 7d avg	Delta
rust-cargo-deny	3m	2m
test-target-determinator	7m	5m

_{settings ⋅ feedback ⋅ docs ⋅ learn more about trunk.io}

[github-actions]

✅ Forge suite realistic_env_max_load success on d6e3f8c61d6c5c42324dd307764f123c05686fe1

two traffics test: inner traffic : committed: 14849.58 txn/s, latency: 2667.48 ms, (p50: 2700 ms, p70: 2700, p90: 2900 ms, p99: 3200 ms), latency samples: 5646160
two traffics test : committed: 99.99 txn/s, latency: 1449.16 ms, (p50: 1400 ms, p70: 1500, p90: 1600 ms, p99: 2800 ms), latency samples: 1800
Latency breakdown for phase 0: ["MempoolToBlockCreation: max: 1.435, avg: 1.407", "ConsensusProposalToOrdered: max: 0.292, avg: 0.289", "ConsensusOrderedToCommit: max: 0.415, avg: 0.399", "ConsensusProposalToCommit: max: 0.705, avg: 0.689"]
Max non-epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 1.32s no progress at version 15421 (avg 0.19s) [limit 15].
Max epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 0.62s no progress at version 2247039 (avg 0.62s) [limit 16].
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions]

✅ Forge suite compat success on 60f7ca8827f5d64a148c3b163dc4126b0879279b ==> d6e3f8c61d6c5c42324dd307764f123c05686fe1

Compatibility test results for 60f7ca8827f5d64a148c3b163dc4126b0879279b ==> d6e3f8c61d6c5c42324dd307764f123c05686fe1 (PR)
1. Check liveness of validators at old version: 60f7ca8827f5d64a148c3b163dc4126b0879279b
compatibility::simple-validator-upgrade::liveness-check : committed: 12494.62 txn/s, latency: 2586.91 ms, (p50: 2600 ms, p70: 2800, p90: 3200 ms, p99: 4400 ms), latency samples: 409620
2. Upgrading first Validator to new version: d6e3f8c61d6c5c42324dd307764f123c05686fe1
compatibility::simple-validator-upgrade::single-validator-upgrading : committed: 4123.88 txn/s, latency: 7589.52 ms, (p50: 8300 ms, p70: 9000, p90: 9200 ms, p99: 9300 ms), latency samples: 89780
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 4238.09 txn/s, latency: 8019.53 ms, (p50: 9000 ms, p70: 9100, p90: 9200 ms, p99: 9400 ms), latency samples: 148760
3. Upgrading rest of first batch to new version: d6e3f8c61d6c5c42324dd307764f123c05686fe1
compatibility::simple-validator-upgrade::half-validator-upgrading : committed: 4300.25 txn/s, latency: 7208.86 ms, (p50: 8000 ms, p70: 8600, p90: 8900 ms, p99: 9000 ms), latency samples: 91720
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 3921.18 txn/s, latency: 8758.81 ms, (p50: 9800 ms, p70: 9900, p90: 10200 ms, p99: 10200 ms), latency samples: 138400
4. upgrading second batch to new version: d6e3f8c61d6c5c42324dd307764f123c05686fe1
compatibility::simple-validator-upgrade::rest-validator-upgrading : committed: 6892.56 txn/s, latency: 4399.52 ms, (p50: 5200 ms, p70: 5300, p90: 5600 ms, p99: 6100 ms), latency samples: 128200
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 6991.95 txn/s, latency: 4862.41 ms, (p50: 5200 ms, p70: 5600, p90: 5800 ms, p99: 6300 ms), latency samples: 236440
5. check swarm health
Compatibility test for 60f7ca8827f5d64a148c3b163dc4126b0879279b ==> d6e3f8c61d6c5c42324dd307764f123c05686fe1 passed
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking