[Bug]: Azure Provider Containerservice does not work after update >= 1.2.0

[Crazyigor1987]

Is there an existing issue for this?

• I have searched the existing issues

Affected Resource(s)

containerservice.azure.upbound.io/v1beta1

Resource MRs required to reproduce the bug

apiVersion: pkg.crossplane.io/v1alpha1
kind: ControllerConfig
metadata:
  name: debug-config
spec:
  args:
    - --debug
---
apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
  name: provider-azure-storage
spec:
  package: xpkg.upbound.io/upbound/provider-azure-storage:v1.3.0
  controllerConfigRef:
    name: debug-config
---
apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
  name: provider-azure-network
spec:
  package: xpkg.upbound.io/upbound/provider-azure-network:v1.3.0
  controllerConfigRef:
    name: debug-config
---
apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
  name: provider-azure-keyvault
spec:
  package: xpkg.upbound.io/upbound/provider-azure-keyvault:v1.3.0
  controllerConfigRef:
    name: debug-config

---
apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
  name: provider-azure-managedidentity
spec:
  package: xpkg.upbound.io/upbound/provider-azure-managedidentity:v1.3.0
  controllerConfigRef:
    name: debug-config

---
apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
  name: provider-azure-compute
spec:
  package: xpkg.upbound.io/upbound/provider-azure-compute:v1.3.0
  controllerConfigRef:
    name: debug-config

---
apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
  name: provider-azure-containerservice
spec:
  package: xpkg.upbound.io/upbound/provider-azure-containerservice:v1.3.0
  controllerConfigRef:
    name: debug-config

---
apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
  name: provider-azure-authorization
spec:
  package: xpkg.upbound.io/upbound/provider-azure-authorization:v1.3.0
  controllerConfigRef:
    name: debug-config

---
apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
  name: upbound-provider-family-azure
spec:
  package: xpkg.upbound.io/upbound/provider-family-azure:v1.3.0
  controllerConfigRef:
    name: debug-config
---
apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
  name: upbound-provider-provider-argocd
spec:
  package: xpkg.upbound.io/crossplane-contrib/provider-argocd:v0.7.0
  controllerConfigRef:
    name: debug-config
---
apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
  name: provider-kubernetes
spec:
  package: xpkg.upbound.io/crossplane-contrib/provider-kubernetes:v0.14.0
  controllerConfigRef:
    name: debug-config

Steps to Reproduce

I have updated the provider configs from version 1.1.0 to 1.3.0.

What happened?

After the update, i am not able to get informations about the running clusters created by the containerservice api.

user@local:~$ kubectl get kubernetesclusters.containerservice.azure.upbound.io 
Error from server: conversion webhook for containerservice.azure.upbound.io/v1beta1, Kind=KubernetesCluster failed: Post "https://provider-azure-containerservice.crossplane.svc:9443/convert?timeout=30s": EOF

Pods and crossplane are running

user@local:~$ kubectl -n crossplane get pods
NAME                                                             READY   STATUS    RESTARTS       AGE
crossplane-785f969dc5-62xmf                                      2/2     Running   0              5d20h
crossplane-785f969dc5-j4bn7                                      2/2     Running   0              5d23h
crossplane-785f969dc5-w6kdz                                      2/2     Running   0              5d23h
crossplane-rbac-manager-84b7bf46db-r6lx8                         2/2     Running   10 (15h ago)   5d23h
provider-azure-authorization-ca718ab078b2-5fc767b8bc-x92mp       2/2     Running   0              6m53s
provider-azure-compute-5f7e12c15a84-776488c659-92lgk             2/2     Running   0              6m45s
provider-azure-containerservice-7521bcd1e3c1-589f7b58f9-rlq62    2/2     Running   0              6m59s
provider-azure-keyvault-5e7163c6af8f-7c4dc77644-z2dvp            2/2     Running   0              7m1s
provider-azure-managedidentity-ef0465e36bb5-646d6767f6-vsqtd     2/2     Running   0              6m47s
provider-azure-network-8ade55f8f04c-657dd6cb67-fprg5             2/2     Running   0              6m43s
provider-azure-storage-a1be2f8ec957-6fff94df49-5d77l             2/2     Running   0              6m51s
provider-kubernetes-f8518c887488-76f4dcb5f4-87c58                2/2     Running   0              5d23h
provider-terraform-ff16e1bc51b9-7d54565d4c-pq529                 2/2     Running   0              5d17h
upbound-provider-family-azure-b3f8f01fe9bd-85f7469fcd-hs8bv      2/2     Running   0              7m2s
upbound-provider-provider-argocd-899b2f59c6aa-7f9585459f-bvfcc   2/2     Running   0              5d21h

Relevant Error Output Snippet

The logs give no indication of misbehaviour.

user@local:~$ kubectl -n crossplane logs provider-azure-containerservice-7521bcd1e3c1-589f7b58f9-rlq62 package-runtime
2024-06-18T11:20:10Z	DEBUG	provider-azure	Starting	{"sync-interval": "1h0m0s", "poll-interval": "10m0s", "poll-jitter": "30s", "max-reconcile-rate": 100}

Crossplane Version

crossplane/crossplane:v1.16.0

Provider Version

1.3.0

Kubernetes Version

v1.27.9

Kubernetes Distribution

AKS

Additional Info

No response

[turkenf]

Hi @Crazyigor1987,

Thank you for the issue report, I'm asking to understand fully. Are you saying that the KubernetesCluster resource that you manage in provider-azure-containerservice v1.1.0 does not work after you upgrade the provider version to 1.3.0?

[turkenf]

Are you experiencing this problem only for the KubernetesCluster resource?

[Crazyigor1987]

Hi @turkenf ,

I've updated the providers to v1.4.0. Experience the problem also on this version. These CRDs are not accessable:

diskencryptionsets.compute.azure.upbound.io:
Error from server: conversion webhook for compute.azure.upbound.io/v1beta1, Kind=DiskEncryptionSet failed: Post "https://provider-azure-compute.crossplane.svc:9443/convert?timeout=30s": EOF

dnszones.network.azure.upbound.io:
Error from server: conversion webhook for network.azure.upbound.io/v1beta1, Kind=DNSZone failed: Post "https://provider-azure-network.crossplane.svc:9443/convert?timeout=30s": EOF

keys.keyvault.azure.upbound.io:
Error from server: conversion webhook for keyvault.azure.upbound.io/v1beta1, Kind=Key failed: Post "https://provider-azure-keyvault.crossplane.svc:9443/convert?timeout=30s": EOF

kubernetesclusternodepools.containerservice.azure.upbound.io:
Error from server: conversion webhook for containerservice.azure.upbound.io/v1beta1, Kind=KubernetesClusterNodePool failed: Post "https://provider-azure-containerservice.crossplane.svc:9443/convert?timeout=30s": EOF

kubernetesclusters.containerservice.azure.upbound.io:
Error from server: conversion webhook for containerservice.azure.upbound.io/v1beta1, Kind=KubernetesCluster failed: Post "https://provider-azure-containerservice.crossplane.svc:9443/convert?timeout=30s": EOF

subnets.network.azure.upbound.io:
Error from server: conversion webhook for network.azure.upbound.io/v1beta1, Kind=Subnet failed: Post "https://provider-azure-network.crossplane.svc:9443/convert?timeout=30s": EOF

vaults.keyvault.azure.upbound.io:
Error from server: conversion webhook for keyvault.azure.upbound.io/v1beta1, Kind=Vault failed: Post "https://provider-azure-keyvault.crossplane.svc:9443/convert?timeout=30s": EOF

virtualnetworks.network.azure.upbound.io:
Error from server: conversion webhook for network.azure.upbound.io/v1beta1, Kind=VirtualNetwork failed: Post "https://provider-azure-network.crossplane.svc:9443/convert?timeout=30s": EOF

[Crazyigor1987]

I got it working after a cleaned reinstall of crossplane and the providers to v.1.5.0. Today i face the same issue after updating it to v1.7.0.

Errors with "kubectl get managed":

Error from server: conversion webhook for compute.azure.upbound.io/v1beta1, Kind=DiskEncryptionSet failed: Post "https://provider-azure-compute.crossplane.svc:9443/convert?timeout=30s": EOF
Error from server: conversion webhook for containerservice.azure.upbound.io/v1beta1, Kind=KubernetesClusterNodePool failed: Post "https://provider-azure-containerservice.crossplane.svc:9443/convert?timeout=30s": net/http: request canceled (Client.Timeout exceeded while awaiting headers)
Error from server: conversion webhook for containerservice.azure.upbound.io/v1beta1, Kind=KubernetesCluster failed: Post "https://provider-azure-containerservice.crossplane.svc:9443/convert?timeout=30s": net/http: request canceled (Client.Timeout exceeded while awaiting headers)

[Crazyigor1987]

Looks like it is related to #784

[github-actions]

This provider repo does not have enough maintainers to address every issue. Since there has been no activity in the last 90 days it is now marked as stale. It will be closed in 14 days if no further activity occurs. Leaving a comment starting with /fresh will mark this issue as not stale.