Having a lot of alerts "KubeCPUOvercommit"

[sneko]

Hey,

It appears I have every day the alert KubeCPUOvercommit but everything seems right.

I'm wondering if I really have an issue that could occur? I'm using an "autoscaler" to populate an additional node in case my CPU/memory requests are too limited. This should be fine no?

I don't know if this alert can be ignored in case of an autoscaler? I didn't see information about this.

Thank you,

[sneko]

@paulfantom is there another place where I can get community help? I did some research for this but unable to find what's wrong.

From what I see:

  - name: kubernetes-resources
    rules:
    - alert: KubeCPUOvercommit
      annotations:
        description: Cluster has overcommitted CPU resource requests for Pods and cannot tolerate node failure.
        runbook_url: https://github.com/prometheus-operator/kube-prometheus/wiki/kubecpuovercommit
        summary: Cluster has overcommitted CPU resource requests.
      expr: |
        sum(namespace_cpu:kube_pod_container_resource_requests:sum{})
          /
        sum(kube_node_status_allocatable{resource="cpu"})
          >
        ((count(kube_node_status_allocatable{resource="cpu"}) > 1) - 1) / count(kube_node_status_allocatable{resource="cpu"})
      for: 5m
      labels:
        severity: warning

According to Grafana at the cluster level I'm only at:

• 64% of "CPU Requests Commitment"
• 770% of "CPU Limits Commitment"

But when receiving the alerts I do see tiny pikes of CPU usage... I don't get it, I'm doing tiny tasks... by why not. What should I do, request more CPU to schedule my pods? to be sure everything is fine?

Thank you,

EDIT: see @paulfantom comment for the explanation (not able to mark a comment of comment as the answer ^^...)

[paulfantom]

The alert itself is a bit more involved than just simple comparing resource requests vs what is available in a cluster. What it does is try to figure out if your cluster can handle node failure based on overall CPU requests. To do so it first calculates CPU occupancy by taking a ratio of all CPU requests to allocatable CPU (LHS of the alert expression), next it compares that ratio to (num_of_nodes - 1) / num_of_nodes (RHS of alert expression).

This works under the assumption that your cluster is homogenous though.

On examples:

• Assuming you have 2 node cluster, RHS is (2-1)/2 = 0.5. If you request more than 50% of the total allocatable CPU pool, the alert will fire.
• Assuming you have 9 node cluster, RHS is (9-1)/9 = 0.89. So if you request more than 89% of the total allocatable CPU pool, the alert will fire.

Back to your case. When you are using cluster autoscaler it should be fine to ignore the alert as long as you can live with some pods being in Pending state for a bit more time.

[sneko]

Thank you @paulfantom for your answer :)

I do have 4 nodes, so triggering at >0.75 . I will see if I silent or not the alert, or maybe adjust CPU request if some pods are not well-aligned.

Following your explanation, I guess it does not take in account node taints (nodeAffinity...), so if I have 1 node dedicated to something specific, the formula becomes wrong, and at the end, is optimistic.

[paulfantom]

Yes, you are correct that it doesn't take into account any taints, zones, affinities, etc. It works under very optimistic assumptions.