ignoreDifferences not working for a openshift machineSet deployed with helm and mutated with Kyverno

[nsalfos]

I think i tried everything but i cannot get THe resource to get created and left alone by ArgoCD.
The machineSet im deploying has the keyword CLUSTERID injected in several places of the manifest.
Kyverno will mutate each instance of the word for the actual cluster name.
if i deploy it using helm install it works great, but when deployed by ArgoCD it gets created > mutated > Out of Sync > Deleted > created. in an endless loop.

i have tried removing all the jsonPointers in order to ingore the whole api-resrouce group but even that didn't work.
and i'd rather just ignore only the lines that will get mutated by Kyverno, aka anything with CLUSTERID on it.

This is the result of running helm install kyverno-machineset -f ../../myvalues.yml ./ocp-machineset --dry-run --debug

# Source: ocp-machineset/templates/machineset.yaml
apiVersion: machine.openshift.io/v1beta1
kind: MachineSet
metadata:
  creationTimestamp: null
  labels:
    machine.openshift.io/cluster-api-cluster: CLUSTERID
  name: CLUSTERID-kyverno-worker
  namespace: openshift-machine-api
spec:
  replicas: 1
  selector:
    matchLabels:
      machine.openshift.io/cluster-api-cluster: CLUSTERID
      machine.openshift.io/cluster-api-machineset: CLUSTERID-kyverno-worker
      
  template:
    metadata:
      creationTimestamp: null
      labels:
        machine.openshift.io/cluster-api-cluster: CLUSTERID
        machine.openshift.io/cluster-api-machine-role: kyverno
        machine.openshift.io/cluster-api-machine-type: kyverno
        machine.openshift.io/cluster-api-machineset: CLUSTERID-kyverno-worker
      
    spec:
      metadata:
      
        labels:
          node-role.kubernetes.io/kyverno: ""
      
    
      taints:
      - key: node-role.kubernetes.io/kyverno
        effect: NoSchedule
    
      providerSpec:
        value:
          apiVersion: vsphereprovider.openshift.io/v1beta1
          credentialsSecret:
            name: vsphere-cloud-credentials
          diskGiB: 200
          kind: VSphereMachineProviderSpec
          memoryMiB: 76000
          metadata:
            creationTimestamp: null
          network:
            devices:
            - networkName: LAB-NET
          numCPUs: 16
        
          numCoresPerSocket: 16
        
          snapshot: ""
          template: CLUSTERID-rhcos
          userDataSecret:
            name: worker-user-data
          workspace:
            datacenter: lab
            datastore: vsphere_openshift
            folder: /lab/folder/to/datastore
            resourcePool: /lab/folder/to/resourcepool
            server: vcenter.lab.example.com

this is my ArgoCD app.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: app-worker-nodes-kyverno-test
  labels:
    clusterName: lab1
    owner: "John Doe"
spec:
  project: project
  source:
    chart: "ocp-machineset"
    repoURL: 'https://repo.example.com'
    targetRevision: ^0.1.1-0
    helm:
      values: |-
        cluster: "CLUSTERID"
        vmware:
          datacenter: "lab"
          datastore: "vsphere_openshift"
          folder: " /lab/folder/to/datastore"
          resourcePool: "/lab/folder/to/resourcepool"
          server: "vcenter.lab.example.com"
        networkName: "LAB-NET"

        machineset:
          name: kyverno
          storage: 200
          cpu: 16
          memory: 76000
          replicas: 1
          isDefaultWorker: false
  destination:
    server: 'https://lab.cluster.example.com'
    namespace: openshift-machine-api
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
    - Replace=true
  resource.compareoptions: |
    # disables status field diffing in specified resource types
    ignoreAggregatedRoles: true
  ignoreDifferences:
  - group: machine.openshift.io
    kind: MachineSet
    namespace: openshift-machine-api
    jsonPointers:
    - /metadata/name
    - /metadata/labels/machine.openshift.io~1cluster-api-cluster
    - /spec/selector/matchLabels/machine.openshift.io~1cluster-api-cluster
    - /spec/selector/matchLabels/machine.openshift.io~1cluster-api-machineset
    - /spec/template/metadata/labels/machine.openshift.io~1cluster-api-cluster
    - /spec/template/metadata/labels/machine.openshift.io~1cluster-api-machineset
    - /spec/template/spec/metadata/providerSpec/value/template

[nastacio]

I have seen this type of problem where ArgoCD creates something that, in turn, creates something else. Sometimes, that "something" propagates its labels to its offspring, including the app.kubernetes.io/instance: _name_of_the_argo_application_ label.

Then Argo looks at that label in the created objects and realizes they are not in the original repo URL and, naturally, decides that these objects are a matter of configuration drift and deletes them.

In this scenario, the IgnoreExtraneous annotation may help:
https://argo-cd.readthedocs.io/en/stable/user-guide/compare-options/

All that said, I am slightly surprised that the resources are getting recreated, but I guess I would have to see the entire repository to understand the cause for that.

Also, if you don't care about managing MachineSet at all, you could set its exclusion globally from the drift detection: https://argo-cd.readthedocs.io/en/stable/operator-manual/declarative-setup/#resource-exclusioninclusion

[nsalfos]

Mmm, not quite what's happening here though.
ArgoCD creates a MachineSet resoource called CLUSTERID-kyverno-worker which contains the manifest previously mentioned. (with all the CLUSTERID keywords) that is actually Mutated by kyverno... so technically is not an "extraneous object" is the same object but now called (assuming the infra id of the cluster is: lab-cluster-abcde ) lab-cluster-abcde-kyverno-worker

and then the manifest looks something like this:

apiVersion: machine.openshift.io/v1beta1
kind: MachineSet
metadata:
  creationTimestamp: null
  labels:
    machine.openshift.io/cluster-api-cluster: lab-cluster-abcde
  name: lab-cluster-abcde-kyverno-worker
  namespace: openshift-machine-api
spec:
  replicas: 1
  selector:
    matchLabels:
      machine.openshift.io/cluster-api-cluster: lab-cluster-abcde
      machine.openshift.io/cluster-api-machineset: lab-cluster-abcde-kyverno-worker
      
  template:
    metadata:
      creationTimestamp: null
      labels:
        machine.openshift.io/cluster-api-cluster: lab-cluster-abcde
        machine.openshift.io/cluster-api-machine-role: kyverno
        machine.openshift.io/cluster-api-machine-type: kyverno
        machine.openshift.io/cluster-api-machineset: lab-cluster-abcde-kyverno-worker
...

So, that makes ArgoCD belive that the resrouce in teh source files is missing, and the actual resrouce present in the cluster needs to be pruned. (image below)

[nsalfos]

I did try adding the argocd.argoproj.io/compare-options: IgnoreExtraneous and also /metadata/annotations/ to the ignoreDifferences.jsonPointers but still the same results.

[Aym3nTN]

Were you able to find a solution this usecase?

[nsalfos]

Hey, not really, cant remember much right now, but i think we gave up on kyverno mutating this stuff.

[Aym3nTN]

Ah same, we ended up using kustomize patch