Stop provider upgrade if clients version is lagging

[leelavg]

Provider supports client operator at X.Y and X.(Y-1) versions and if any client is already at X.(Y-1) version and provider gets upgraded then the lagging client goes out of support

This PR prohibits such provider upgrades

Minor: use a single existing enqueue function and removes duplicates

[RHSTOR-5073]

[leelavg]

/hold

1. More testing and fixing existing tests in under progress
2. Will unhold when more than one vote is provided
3. Testing done so far is as below

=== Provider Operator is at 4.15
# ko get csv -ojsonpath='{range .items[*]}[{.metadata.name}, {.spec.version}]{end}'
[ocs-operator.v4.15.0-79.stable, 4.15.0-79.stable]

=== All connected clients are also at 4.15
# ko get storageconsumer -ojsonpath='{range .items[*]}[{.metadata.name}, {.status.client.operatorVersion}]{"\n"}{end}'
[cons-1, 4.15.1]
[cons-2, 4.15.2]
[cons-3, 4.15.3]

=== Provider can upgrade w/o any issue
# ko get condition -ojsonpath='{range .items[*]}[{.metadata.name}, {.spec.conditions}]{end}' 
[ocs-operator.v4.15.0-79.stable, [{"lastTransitionTime":"2023-12-07T13:57:06Z","message":"Reconcile completed successfully","reason":"ReconcileCompleted","status":"True","type":"Upgradeable"}]]

==== Assume one of the client operator is lagging provider operator by 1 minor version
# ko patch --subresource=status storageconsumers cons-3 --type json -p '[{"op": "replace", "path": "/status/client/operatorVersion", "value": "4.14.3"}]'
storageconsumer.ocs.openshift.io/cons-3 patched

# ko get storageconsumer -ojsonpath='{range .items[*]}[{.metadata.name}, {.status.client.operatorVersion}]{"\n"}{end}'
[cons-1, 4.15.1]
[cons-2, 4.15.2]
[cons-3, 4.14.3]

==== The scenario is surfaced to the Operator Condition and upgrade is prohibited
# ko get condition -ojsonpath='{range .items[*]}[{.metadata.name}, {.spec.conditions}]{end}' 
[ocs-operator.v4.15.0-79.stable, [{"lastTransitionTime":"2023-12-07T13:58:10Z","message":"StorageConsumer \"cons-3\" will become unsupported if upgraded","reason":"ConsumerClientOperatorVersion","status":"False","type":"Upgradeable"}]]

# ko get storagecluster
NAME                 AGE   PHASE   EXTERNAL   CREATED AT             VERSION
ocs-storagecluster   30h   Ready              2023-12-06T07:20:31Z   4.15.0

==== Reset client operator version
# ko patch --subresource=status storageconsumers cons-3 --type json -p '[{"op": "replace", "path": "/status/client/operatorVersion", "value": "4.15.3"}]'
storageconsumer.ocs.openshift.io/cons-3 patched

# ko get storageconsumer -ojsonpath='{range .items[*]}[{.metadata.name}, {.status.client.operatorVersion}]{"\n"}{end}'
[cons-1, 4.15.1]
[cons-2, 4.15.2]
[cons-3, 4.15.3]

==== If the connected client operator is upto w/ provider operator upgrade will be unblocked
# ko get condition -ojsonpath='{range .items[*]}[{.metadata.name}, {.spec.conditions}]{end}' 
[ocs-operator.v4.15.0-79.stable, [{"lastTransitionTime":"2023-12-07T13:59:11Z","message":"Reconcile completed successfully","reason":"ReconcileCompleted","status":"True","type":"Upgradeable"}]]

[leelavg]

depends on #2308 partially

[leelavg]

rebased

[leelavg]

address comments and retested.

[leelavg]

/unhold

[nb-ohad]

/lgtm

[leelavg]

rebased.

[nb-ohad]

/lgtm

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: leelavg, nb-ohad

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [nb-ohad]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[leelavg]

/cherrypick release-4.15

[openshift-cherrypick-robot]

@leelavg: new pull request created: #2368

In response to this:

/cherrypick release-4.15

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.