fix(k8s-eks): use latest haproxy ingress controller version

[vponomaryov]

With the scylla-operator-1.10 we started getting HA problems [1] with the haproxy service.
So, fix it by using the latest available ingress controller version for haproxy.

[1] scylladb/scylla-operator#1341

PR pre-checks (self review)

• I followed KISS principle and best practices
• I didn't leave commented-out/debugging code
• I added the relevant backport labels
• New configuration option are added and documented (in sdcm/sct_config.py)
• I have added tests to cover my changes (Infrastructure only - under unit-test/ folder)
• All new and existing unit tests passed (CI)
• I have updated the Readme/doc folder accordingly (if needed)

[vponomaryov]

Without this PR: https://jenkins.scylladb.com/job/scylla-operator/job/operator-1.10/job/upgrade/job/upgrade-platform-k8s-eks/3/consoleFull
With this PR: https://jenkins.scylladb.com/view/staging/job/scylla-staging/job/valerii/job/vp-upgrade-platform-k8s-eks/128/consoleFull

[soyacz]

LGTM
Question: how we assure we use the same as on production?

[vponomaryov]

LGTM Question: how we assure we use the same as on production?

We have used the same.
They use image: haproxytech/kubernetes-ingress@sha256:324efe481132105631e12e5ab144d38dd533425705daa76cbbc78cdb91d8a662
We use 1.9.0.
It is the same: https://hub.docker.com/layers/haproxytech/kubernetes-ingress/1.9.0/images/sha256-324efe481132105631e12e5ab144d38dd533425705daa76cbbc78cdb91d8a662?context=explore

But, the latest one is more stable. And I would say that they should switch to the latest too.
I think no one updated it there just because of the busyness with other stuff.

And, finally, the haproxy solution will be replaced soon with the self-baked solution with required whistles which are absent in haproxy. So, we are ok to just make it more stable for now without big time investment from our side.

[vponomaryov]

One more run showed that the problem is still in place: https://jenkins.scylladb.com/job/scylla-operator/job/operator-1.10/job/upgrade/job/upgrade-platform-k8s-eks/4/consoleFull
So, for now, with this latest version we have 1 failed run and 1 passed.

[vponomaryov]

Third run with this PR passed: https://jenkins.scylladb.com/job/scylla-operator/job/operator-1.10/job/upgrade/job/upgrade-platform-k8s-eks/5/
So, I consider it a pretty good results that 2 out of 3 runs pass when with old version 2 out of 2 failed.

[fruch]

LGTM