Relax validation rules enforcing immutability of replicated jobs

[danielvegamyhre]

What would you like to be added:
Relax validation rules enforcing immutability of replicated jobs. Specifically:

• .spec.replicatedJobs[*].spec.template.spec.template (job podTemplateSpec, to add annotations, taint tolerations, etc)

Why is this needed:
DWS compatibility with JobSet. cc @alculquicondor for more context

[alculquicondor]

When using ResourceFlavors with nodeLabels, Kueue will inject node selectors to the Pod.

When using cluster-autoscaler ProvisioningRequest, Kueue will inject an annotation to the Pod template.

These fields should be mutable, especially if the Jobset is suspended or is just about to transition to suspend=false.

[danielvegamyhre]

Planning on including this in a patch release early next week. cc @ahg-g

I think a patch release is fine, since the validation is being relaxed (not increased) so we won't break any running workloads running on v0.5.x and upgrading to this patch (v0.5.2).

Alternatively we can include this in v0.6.0, which will be ready soon as well, after we do some manual testing of #537 (targeting June 1st).

[kannon92]

I would vote to put it in a new release. It’s not really something I would think to be included in a patch release.

[danielvegamyhre]

Reduced scope of bug specifically to replicatedJob pod template since #529 already exists

[danielvegamyhre]

I would vote to put it in a new release. It’s not really something I would think to be included in a patch release.

I reduced the scope of the changes to be the validation fix only, and postponed the elastic jobset support to be included in v0.6.0. So I think we can go ahead and do a patch release for this.