kube-state-metrics doesn't work

[ibadullaev-inc4]

Hi I installed k8s cluster version AMS3 - 1.27.2-do.0
After I install kube-state-metrics via Marketplace 1CLICK APPS

But metrics api doesn't work

[nariman@notebook k8s]$ kubectl top nodes
error: Metrics API not available

[nariman@notebook k8s]$ helm ls -n kube-system
NAME              	NAMESPACE  	REVISION	UPDATED                                	STATUS  	CHART                   	APP VERSION
kube-state-metrics	kube-system	1       	2023-06-28 12:13:08.794775262 +0000 UTC	deployed	kube-state-metrics-3.2.0	2.5.0 

[nariman@notebook k8s]$ kubectl get apiservices.apiregistration.k8s.io 
NAME                                   SERVICE   AVAILABLE   AGE
v1.                                    Local     True        6d5h
v1.admissionregistration.k8s.io        Local     True        6d5h
v1.apiextensions.k8s.io                Local     True        6d5h
v1.apps                                Local     True        6d5h
v1.authentication.k8s.io               Local     True        6d5h
v1.authorization.k8s.io                Local     True        6d5h
v1.autoscaling                         Local     True        6d5h
v1.batch                               Local     True        6d5h
v1.certificates.k8s.io                 Local     True        6d5h
v1.coordination.k8s.io                 Local     True        6d5h
v1.discovery.k8s.io                    Local     True        6d5h
v1.events.k8s.io                       Local     True        6d5h
v1.networking.k8s.io                   Local     True        6d5h
v1.node.k8s.io                         Local     True        6d5h
v1.policy                              Local     True        6d5h
v1.rbac.authorization.k8s.io           Local     True        6d5h
v1.scheduling.k8s.io                   Local     True        6d5h
v1.security.istio.io                   Local     True        20m
v1.snapshot.storage.k8s.io             Local     True        20m
v1.storage.k8s.io                      Local     True        6d5h
v1alpha1.extensions.istio.io           Local     True        20m
v1alpha1.install.istio.io              Local     True        20m
v1alpha1.telemetry.istio.io            Local     True        20m
v1alpha3.networking.istio.io           Local     True        20m
v1beta1.networking.istio.io            Local     True        20m
v1beta1.security.istio.io              Local     True        20m
v1beta2.flowcontrol.apiserver.k8s.io   Local     True        6d5h
v1beta3.flowcontrol.apiserver.k8s.io   Local     True        6d5h
v2.autoscaling                         Local     True        6d5h
v2.cilium.io                           Local     True        20m
v2alpha1.cilium.io                     Local     True        20m

[nariman@notebook k8s]$ kubectl -n kube-system get deployments.apps kube-state-metrics -o yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "1"
    meta.helm.sh/release-name: kube-state-metrics
    meta.helm.sh/release-namespace: kube-system
  creationTimestamp: "2023-06-28T12:13:11Z"
  generation: 1
  labels:
    app.kubernetes.io/instance: kube-state-metrics
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: kube-state-metrics
    helm.sh/chart: kube-state-metrics-3.2.0
  name: kube-state-metrics
  namespace: kube-system
  resourceVersion: "2299134"
  uid: f477f062-08ed-4c36-bae7-f4c6e1c725a0
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app.kubernetes.io/instance: kube-state-metrics
      app.kubernetes.io/name: kube-state-metrics
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        app.kubernetes.io/instance: kube-state-metrics
        app.kubernetes.io/managed-by: Helm
        app.kubernetes.io/name: kube-state-metrics
        helm.sh/chart: kube-state-metrics-3.2.0
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - podAffinityTerm:
              labelSelector:
                matchLabels:
                  app.kubernetes.io/instance: kube-state-metrics
                  app.kubernetes.io/name: kube-state-metrics
              namespaces:
              - kube-system
              topologyKey: kubernetes.io/hostname
            weight: 1
      containers:
      - args:
        - --resources=certificatesigningrequests
        - --resources=configmaps
        - --resources=cronjobs
        - --resources=daemonsets
        - --resources=deployments
        - --resources=endpoints
        - --resources=horizontalpodautoscalers
        - --resources=ingresses
        - --resources=jobs
        - --resources=limitranges
        - --resources=mutatingwebhookconfigurations
        - --resources=namespaces
        - --resources=networkpolicies
        - --resources=nodes
        - --resources=persistentvolumeclaims
        - --resources=persistentvolumes
        - --resources=poddisruptionbudgets
        - --resources=pods
        - --resources=replicasets
        - --resources=replicationcontrollers
        - --resources=resourcequotas
        - --resources=secrets
        - --resources=services
        - --resources=statefulsets
        - --resources=storageclasses
        - --resources=volumeattachments
        image: docker.io/bitnami/kube-state-metrics:2.5.0-debian-11-r23
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 6
          httpGet:
            path: /healthz
            port: http
            scheme: HTTP
          initialDelaySeconds: 120
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 5
        name: kube-state-metrics
        ports:
        - containerPort: 8080
          name: http
          protocol: TCP
        readinessProbe:
          failureThreshold: 6
          httpGet:
            path: /
            port: http
            scheme: HTTP
          initialDelaySeconds: 30
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 5
        resources: {}
        securityContext:
          runAsNonRoot: true
          runAsUser: 1001
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext:
        fsGroup: 1001
      serviceAccount: kube-state-metrics
      serviceAccountName: kube-state-metrics
      terminationGracePeriodSeconds: 30
status:
  availableReplicas: 1
  conditions:
  - lastTransitionTime: "2023-06-28T12:13:51Z"
    lastUpdateTime: "2023-06-28T12:13:51Z"
    message: Deployment has minimum availability.
    reason: MinimumReplicasAvailable
    status: "True"
    type: Available
  - lastTransitionTime: "2023-06-28T12:13:11Z"
    lastUpdateTime: "2023-06-28T12:13:51Z"
    message: ReplicaSet "kube-state-metrics-6456cbff44" has successfully progressed.
    reason: NewReplicaSetAvailable
    status: "True"
    type: Progressing
  observedGeneration: 1
  readyReplicas: 1
  replicas: 1
  updatedReplicas: 1

[nariman@notebook k8s]$ kubectl -n kube-system get deployments.apps kube-state-metrics 
NAME                 READY   UP-TO-DATE   AVAILABLE   AGE
kube-state-metrics   1/1     1            1           11m

LOGS:

I0628 12:13:12.323524       1 server.go:100] Using resources certificatesigningrequests,configmaps,cronjobs,daemonsets,deployments,endpoints,horizontalpodautoscalers,ingresses,jobs,limitranges,mutatingwebhookconfigurations,namespaces,networkpolicies,nodes,persistentvolumeclaims,persistentvolumes,poddisruptionbudgets,pods,replicasets,replicationcontrollers,resourcequotas,secrets,services,statefulsets,storageclasses,volumeattachments
I0628 12:13:12.325852       1 types.go:136] Using all namespace
I0628 12:13:12.325890       1 server.go:122] Metric allow-denylisting: Excluding the following lists that were on denylist: 
W0628 12:13:12.326204       1 client_config.go:617] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0628 12:13:12.327434       1 server.go:250] Testing communication with server
I0628 12:13:12.345355       1 server.go:255] Running with Kubernetes cluster version: v1.27. git version: v1.27.2. git tree state: clean. commit: 7f6f68fdabc4df88cfea2dcf9a19b2b830f1e647. platform: linux/amd64
I0628 12:13:12.345400       1 server.go:257] Communication with server successful
I0628 12:13:12.346476       1 server.go:202] Starting metrics server: [::]:8080
I0628 12:13:12.346519       1 metrics_handler.go:96] Autosharding disabled
I0628 12:13:12.347392       1 server.go:191] Starting kube-state-metrics self metrics server: [::]:8081
I0628 12:13:12.351414       1 server.go:66] levelinfomsgTLS is disabled.http2false
I0628 12:13:12.351733       1 server.go:66] levelinfomsgTLS is disabled.http2false
I0628 12:13:12.353182       1 builder.go:232] Active resources: certificatesigningrequests,configmaps,cronjobs,daemonsets,deployments,endpoints,horizontalpodautoscalers,ingresses,jobs,limitranges,mutatingwebhookconfigurations,namespaces,networkpolicies,nodes,persistentvolumeclaims,persistentvolumes,poddisruptionbudgets,pods,replicasets,replicationcontrollers,resourcequotas,secrets,services,statefulsets,storageclasses,volumeattachments
W0628 12:13:12.361207       1 reflector.go:324] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167: failed to list *v2beta2.HorizontalPodAutoscaler: the server could not find the requested resource
E0628 12:13:12.361260       1 reflector.go:138] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *v2beta2.HorizontalPodAutoscaler: failed to list *v2beta2.HorizontalPodAutoscaler: the server could not find the requested resource
W0628 12:13:13.367132       1 reflector.go:324] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167: failed to list *v2beta2.HorizontalPodAutoscaler: the server could not find the requested resource
E0628 12:13:13.367211       1 reflector.go:138] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *v2beta2.HorizontalPodAutoscaler: failed to list *v2beta2.HorizontalPodAutoscaler: the server could not find the requested resource
W0628 12:13:15.095647       1 reflector.go:324] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167: failed to list *v2beta2.HorizontalPodAutoscaler: the server could not find the requested resource
E0628 12:13:15.095829       1 reflector.go:138] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *v2beta2.HorizontalPodAutoscaler: failed to list *v2beta2.HorizontalPodAutoscaler: the server could not find the requested resource
W0628 12:13:19.274947       1 reflector.go:324] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167: failed to list *v2beta2.HorizontalPodAutoscaler: the server could not find the requested resource
E0628 12:13:19.274981       1 reflector.go:138] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *v2beta2.HorizontalPodAutoscaler: failed to list *v2beta2.HorizontalPodAutoscaler: the server could not find the requested resource
W0628 12:13:30.730652       1 reflector.go:324] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167: failed to list *v2beta2.HorizontalPodAutoscaler: the server could not find the requested resource
E0628 12:13:30.730704       1 reflector.go:138] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *v2beta2.HorizontalPodAutoscaler: failed to list *v2beta2.HorizontalPodAutoscaler: the server could not find the requested resource
W0628 12:13:47.586634       1 reflector.go:324] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167: failed to list *v2beta2.HorizontalPodAutoscaler: the server could not find the requested resource
E0628 12:13:47.586678       1 reflector.go:138] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *v2beta2.HorizontalPodAutoscaler: failed to list *v2beta2.HorizontalPodAutoscaler: the server could not find the requested resource
W0628 12:14:33.949789       1 reflector.go:324] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167: failed to list *v2beta2.HorizontalPodAutoscaler: the server could not find the requested resource
E0628 12:14:33.949840       1 reflector.go:138] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *v2beta2.HorizontalPodAutoscaler: failed to list *v2beta2.HorizontalPodAutoscaler: the server could not find the requested resource
W0628 12:15:14.552176       1 reflector.go:324] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167: failed to list *v2beta2.HorizontalPodAutoscaler: the server could not find the requested resource
E0628 12:15:14.552210       1 reflector.go:138] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *v2beta2.HorizontalPodAutoscaler: failed to list *v2beta2.HorizontalPodAutoscaler: the server could not find the requested resource
W0628 12:15:50.938306       1 reflector.go:324] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167: failed to list *v2beta2.HorizontalPodAutoscaler: the server could not find the requested resource
E0628 12:15:50.938360       1 reflector.go:138] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *v2beta2.HorizontalPodAutoscaler: failed to list *v2beta2.HorizontalPodAutoscaler: the server could not find the requested resource
W0628 12:16:26.473813       1 reflector.go:324] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167: failed to list *v2beta2.HorizontalPodAutoscaler: the server could not find the requested resource
E0628 12:16:26.480137       1 reflector.go:138] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *v2beta2.HorizontalPodAutoscaler: failed to list *v2beta2.HorizontalPodAutoscaler: the server could not find the requested resource
W0628 12:17:25.972138       1 reflector.go:324] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167: failed to list *v2beta2.HorizontalPodAutoscaler: the server could not find the requested resource
E0628 12:17:25.972247       1 reflector.go:138] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *v2beta2.HorizontalPodAutoscaler: failed to list *v2beta2.HorizontalPodAutoscaler: the server could not find the requested resource
W0628 12:18:17.221516       1 reflector.go:324] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167: failed to list *v2beta2.HorizontalPodAutoscaler: the server could not find the requested resource
E0628 12:18:17.221635       1 reflector.go:138] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *v2beta2.HorizontalPodAutoscaler: failed to list *v2beta2.HorizontalPodAutoscaler: the server could not find the requested resource
W0628 12:19:06.234797       1 reflector.go:324] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167: failed to list *v2beta2.HorizontalPodAutoscaler: the server could not find the requested resource
E0628 12:19:06.234841       1 reflector.go:138] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *v2beta2.HorizontalPodAutoscaler: failed to list *v2beta2.HorizontalPodAutoscaler: the server could not find the requested resource
W0628 12:19:55.369741       1 reflector.go:324] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167: failed to list *v2beta2.HorizontalPodAutoscaler: the server could not find the requested resource
E0628 12:19:55.369772       1 reflector.go:138] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *v2beta2.HorizontalPodAutoscaler: failed to list *v2beta2.HorizontalPodAutoscaler: the server could not find the requested resource
W0628 12:20:47.547929       1 reflector.go:324] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167: failed to list *v2beta2.HorizontalPodAutoscaler: the server could not find the requested resource
E0628 12:20:47.547983       1 reflector.go:138] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *v2beta2.HorizontalPodAutoscaler: failed to list *v2beta2.HorizontalPodAutoscaler: the server could not find the requested resource
W0628 12:21:39.150949       1 reflector.go:324] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167: failed to list *v2beta2.HorizontalPodAutoscaler: the server could not find the requested resource
E0628 12:21:39.150995       1 reflector.go:138] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *v2beta2.HorizontalPodAutoscaler: failed to list *v2beta2.HorizontalPodAutoscaler: the server could not find the requested resource
W0628 12:22:29.390965       1 reflector.go:324] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167: failed to list *v2beta2.HorizontalPodAutoscaler: the server could not find the requested resource
E0628 12:22:29.391096       1 reflector.go:138] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *v2beta2.HorizontalPodAutoscaler: failed to list *v2beta2.HorizontalPodAutoscaler: the server could not find the requested resource
W0628 12:23:18.409274       1 reflector.go:324] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167: failed to list *v2beta2.HorizontalPodAutoscaler: the server could not find the requested resource
E0628 12:23:18.409320       1 reflector.go:138] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *v2beta2.HorizontalPodAutoscaler: failed to list *v2beta2.HorizontalPodAutoscaler: the server could not find the requested resource
W0628 12:24:15.320516       1 reflector.go:324] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167: failed to list *v2beta2.HorizontalPodAutoscaler: the server could not find the requested resource
E0628 12:24:15.320597       1 reflector.go:138] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *v2beta2.HorizontalPodAutoscaler: failed to list *v2beta2.HorizontalPodAutoscaler: the server could not find the requested resource
W0628 12:24:57.078991       1 reflector.go:324] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167: failed to list *v2beta2.HorizontalPodAutoscaler: the server could not find the requested resource
E0628 12:24:57.079031       1 reflector.go:138] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *v2beta2.HorizontalPodAutoscaler: failed to list *v2beta2.HorizontalPodAutoscaler: the server could not find the requested resource
W0628 12:25:39.852846       1 reflector.go:324] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167: failed to list *v2beta2.HorizontalPodAutoscaler: the server could not find the requested resource
E0628 12:25:39.852880       1 reflector.go:138] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *v2beta2.HorizontalPodAutoscaler: failed to list *v2beta2.HorizontalPodAutoscaler: the server could not find the requested resource