DNSPolicy scale test

[mikenairn]

Adds a DNSPolicy specific scale test using kube burner.

Part of #928

Based on the existing scale test, but with a focus on DNSPolicy and shared hostnames being updated by multiple dns operator instances.

The workload will create multiple instances of the dns operator in separate namespaces(kuadrant-dns-operator-x), and multiple test namespaces (scale-test-x) that the corresponding dns operator is configured to watch. The number of dns operator instances and test namespaces created is determined by the JOB_ITERATIONS environment variable.
In each test namespace a test app and service is deployed and one or more gateways are created determined by the NUM_GWS environment variable. The number of listeners added to the gateway is determined by the NUM_LISTENERS environment variable.
Each listener hostname is generated using the listener number and the KUADRANT_ZONE_ROOT_DOMAIN environment variable. In each test namespace a dns provider credential is created, the type created is determined by the DNS_PROVIDER environment variable, additional environment variables may need to be set depending on the provider type.

Requires:

• Add scale test dns-operator#326
• Allow setting metallb CIDR value for local env kuadrant-operator#1110

Comments/Thoughts:

• Kubeburner does not have the concept of running workloads across multiple instances. This was one of the asks in this issue. It is probably possible to run multiple kubeburner tasks simultaneously using the same configuration in order to have multiple updates to the same record set from multiple clusters but there would be no orchestration from kubeburners POV. It should also use of a single thanos instance instead of one deployed on each cluster.
• For these workloads to be of any use we need good metrics and alerts that are expected to fire when things are not working. It's not a test suite with assertions on the state, but rather it expects alerts to fire in order to fail the test run.
• Separating the DNS Operator specific templates/metrics/alerts into the dns operator repo makes sense as long as we have a similar scale test in that repo. TBD if we do want that.

Alerts
A small list of alerts that i realised would be useful, but really there are probably hundreds required.

• Alert when a gateway has not been assigned an address in an appropriate amount of time (Can be hit quite easily when using kind if you only have a few IPs available). This isn't strictly a kuadrant, issue.
• Alert when DNSRecords are in a failing state for a given amount of time.
• Alert if the managers are restarting an unexpected amount of times during the test run. Hit this as part of the DNSRecord scale test, wrote an alert for this here.

[maleck13]

I didn't actually try this and would prefer if @trepel or another of the QE team took a look and approved, but the changes look good to me

[trepel]

I tried against OCP cluster and Route53 and it worked as described - except for that missing github.com/kuadrant/dns-operator/config/observability?ref=main but that's being discussed. What surprises me was that no matter how many GWs/HTTPRoutes are created there is just NUM_LISTENERS unique hostnames. That's by design if I got it right, and it seems to be working (checked by dig +short )

[maleck13]

@mikenairn do we want to move this to ready and get it merged ? @trepel from your comment seems good to merge?

[mikenairn]

What surprises me was that no matter how many GWs/HTTPRoutes are created there is just NUM_LISTENERS unique hostnames.

This is intentional for the workload being added since it's testing dns at scale, part of that is testing that multiple records all contributing to the same dns name works with increasing numbers of owners.

[maleck13]

@trepel are you ok to approve and merge this?

[trepel]

Sorry, this managed to get buried in my todo list. Yes, I am fine to merge this one