Explain some quirks with clusterCIDRIPv4 for Calico CNI

[moshiurH]

Testing for the Calico CIDR option (see sample yaml at https://docs.mirantis.com/mke-docs/docs/concepts/cni/#providers-configuration) was done. It was found that when configuring the Calico spec.network.providers.extraConfig.clusterCIDRIPv4 value, it is easily configurable before creating the cluster. However when trying to modify the clusterCIDRIPv4 value after the cluster has been created, it does not work. Based on Calico's official documentation, it sounds like or is implied that the clusterCIDRIPv4 can be modified via some sort of migration strategy (by creating a new IPPool - basically a rabbit hole).

For the sake of simplicity, maybe we should state clusterCIDRIPv4 can be configured before deployment. For modifying clusterCIDRIPv4 after cluster deployment, consult Mirantis Support.

Some Test Notes:

Case 1 - Modify a default mke4.yaml's:

$ cat mke4.yaml
apiVersion: mke.mirantis.com/v1alpha1
kind: MkeConfig
...
    providers:
    - enabled: true
      extraConfig:
        CALICO_DISABLE_FILE_LOGGING: "true"
        CALICO_STARTUP_LOGLEVEL: DEBUG
        FELIX_LOGSEVERITYSCREEN: DEBUG
        clusterCIDRIPv4: 10.100.0.0/16 #originally 192.168.0.0/16
        enableWireguard: "false"
        ipAutodetectionMethod: ""
        mode: vxlan
        overlay: Always
        vxlanPort: "4789"
        vxlanVNI: "10000"
      provider: calico
...

Deploy with mkectl apply.

Confirm:

$ export KUBECONFIG=./mke.kubeconf
$ calicoctl get ippool -o wide --allow-version-mismatch
NAME                  CIDR            NAT    IPIPMODE   VXLANMODE   DISABLED   DISABLEBGPEXPORT   SELECTOR   
default-ipv4-ippool   10.100.0.0/16   true   Never      Always      false      false              all()      
$ kubectl get ippools.projectcalico.org default-ipv4-ippool -o yaml
apiVersion: projectcalico.org/v3
kind: IPPool
metadata:
  creationTimestamp: "2025-01-06T21:47:57Z"
  generation: 1
  labels:
    app.kubernetes.io/managed-by: tigera-operator
  name: default-ipv4-ippool
  resourceVersion: "2037"
  uid: 48a009df-febe-4280-ab7c-111c93d34558
spec:
  allowedUses:
  - Workload
  - Tunnel
  blockSize: 26
  cidr: 10.100.0.0/16
  ipipMode: Never
  natOutgoing: true
  nodeSelector: all()
  vxlanMode: Always

---

Case 2 - Deploy a default mke4.yaml then reapply with new CIDR:

$ cat mke4.yaml
apiVersion: mke.mirantis.com/v1alpha1
kind: MkeConfig
...
    providers:
    - enabled: true
      extraConfig:
        CALICO_DISABLE_FILE_LOGGING: "true"
        CALICO_STARTUP_LOGLEVEL: DEBUG
        FELIX_LOGSEVERITYSCREEN: DEBUG
        clusterCIDRIPv4: 192.168.0.0/16
        enableWireguard: "false"
        ipAutodetectionMethod: ""
        mode: vxlan
        overlay: Always
        vxlanPort: "4789"
        vxlanVNI: "10000"
      provider: calico
...

Deploy with mkectl apply.

Change clusterCIDRIPv4: 192.168.0.0/16 to clusterCIDRIPv4: 10.100.0.0/16 then run mkectl apply again.

Verify if the CIDR has updated:

$ calicoctl get ippool -o wide --allow-version-mismatch
NAME                  CIDR             NAT    IPIPMODE   VXLANMODE   DISABLED   DISABLEBGPEXPORT   SELECTOR   
default-ipv4-ippool   192.168.0.0/16   true   Never      Always      false      false              all()      
$ kubectl get ippools.projectcalico.org default-ipv4-ippool -o yaml
apiVersion: projectcalico.org/v3
kind: IPPool
metadata:
  creationTimestamp: "2025-01-06T22:49:08Z"
  generation: 1
  labels:
    app.kubernetes.io/managed-by: tigera-operator
  name: default-ipv4-ippool
  resourceVersion: "2006"
  uid: 9888d319-6949-405c-85db-98d4eaa78fa3
spec:
  allowedUses:
  - Workload
  - Tunnel
  blockSize: 26
  cidr: 192.168.0.0/16
  ipipMode: Never
  natOutgoing: true
  nodeSelector: all()
  vxlanMode: Always
$ mkectl config get
apiVersion: mke.mirantis.com/v1alpha1
kind: MkeConfig
...
    providers:
    - enabled: true
      extraConfig:
        CALICO_DISABLE_FILE_LOGGING: "true"
        CALICO_STARTUP_LOGLEVEL: DEBUG
        FELIX_LOGSEVERITYSCREEN: DEBUG
        clusterCIDRIPv4: 10.100.0.0/16
        enableWireguard: "false"
        ipAutodetectionMethod: ""
        mode: vxlan
        overlay: Always
        vxlanPort: "4789"
        vxlanVNI: "10000"
      provider: calico
    - enabled: false
      provider: custom

Notice how mkectl believe that the clusterCIDRIPv4 value is 10.100.0.0/16 but calicoctl still has it set to 192.168.0.0/16.