Split recovery into phases to use parallel steps better

[sshutovskyi-f3]

What problem does this PR solve?

1. I would like to run DR test for blue and green clusters at the same time, for that I need to make sure that recovery only affects the correct cluster
2. Sometimes after the host recovers, the VM is not immediately shown as stopped, so it's not started automatically and k8s node never reaches Ready state, for that VM start - Node uncordon cycle should be retried until all nodes are ready and uncordoned
3. Hosts in inactive clusters are unnecessarily stopped and recovered

What's changed and how it works?

• Only VMs and Nodes from the started host should be checked and recovered
• VM and nodes recovery should be retried multiple times
• Only hosts hosting schedulable k8s nodes are used for chaos

Related changes

• This change also requires further updates to the website (e.g. docs)
• This change also requires further updates to the UI interface

Cherry-pick to release branches (optional)

This PR should be cherry-picked to the following release branches:

• release-2.6
• release-2.5

Checklist

CHANGELOG

Must include at least one of them.

• I have updated the CHANGELOG.md
• I have labeled this PR with "no-need-update-changelog"

Tests

Must include at least one of them.

• Unit test
• E2E test
• Manual test

Side effects

• Breaking backward compatibility

DCO

If you find the DCO check fails, please run commands like below (Depends on the actual situations. For example, if the failed commit isn't the most recent) to fix it:

git commit --amend --signoff
git push --force

[github-actions]

Created new release based on commit 80367d3

Release tag: v2.6.1-f3-80367-sergii-add-more-retries-to-hv-recovery

Link to release

[github-actions]

Created new release based on commit 130c31c

Release tag: v2.6.1-f3-130c3-sergii-add-more-retries-to-hv-recovery

Link to release

[github-actions]

Created new release based on commit b5e7358

Release tag: v2.6.1-f3-b5e73-sergii-add-more-retries-to-hv-recovery

Link to release

[github-actions]

Created new release based on commit 342712c

Release tag: v2.6.1-f3-34271-sergii-add-more-retries-to-hv-recovery

Link to release

[github-actions]

Created new release based on commit 47223fd

Release tag: v2.6.1-f3-47223-sergii-add-more-retries-to-hv-recovery

Link to release

[github-actions]

Created new release based on commit 799001d

Release tag: v2.6.1-f3-79900-sergii-add-more-retries-to-hv-recovery

Link to release

[github-actions]

Created new release based on commit 36ee8e4

Release tag: v2.6.1-f3-36ee8-sergii-add-more-retries-to-hv-recovery

Link to release

[github-actions]

Created new release based on commit ee572ea

Release tag: v2.6.1-f3-ee572-sergii-add-more-retries-to-hv-recovery

Link to release

[github-actions]

Created new release based on commit 22005fc

Release tag: v2.6.1-f3-22005-sergii-add-more-retries-to-hv-recovery

Link to release

[github-actions]

Created new release based on commit 8d60753

Release tag: v2.6.1-f3-8d607-sergii-add-more-retries-to-hv-recovery

Link to release

[github-actions]

Created new release based on commit 986dcf8

Release tag: v2.6.1-f3-986dc-sergii-add-more-retries-to-hv-recovery

Link to release

[github-actions]

Created new release based on commit 31948a7

Release tag: v2.6.1-f3-31948-sergii-add-more-retries-to-hv-recovery

Link to release

[github-actions]

Created new release based on commit 1d67718

Release tag: v2.6.1-f3-1d677-sergii-add-more-retries-to-hv-recovery

Link to release

[github-actions]

Created new release based on commit 13b84d7

Release tag: v2.6.1-f3-13b84-sergii-add-more-retries-to-hv-recovery

Link to release

[github-actions]

Created new release based on commit 50866bf

Release tag: v2.6.1-f3-50866-sergii-add-more-retries-to-hv-recovery

Link to release

[github-actions]

Created new release based on commit ef32cdc

Release tag: v2.6.1-f3-ef32c-sergii-add-more-retries-to-hv-recovery

Link to release