Several components alerting as down, kubecontrollermanager, kubescheduler.

[jeanluclariviere]

What did you do?
Deployed kube-prometheus on a kubeadm cluster with pretty much default settings and only a few additional libraries enabled.

Did you expect to see some different?
After deploying the manifests, I can see the following alerts firing:

KubeControllerManagerDown
KubeSchedulerDown
TargetDown (kube-state-metrics/ down)

• Prometheus Operator version:
  prometheus-operator:v0.42.1

• Kubernetes version information:
  Running release-0.6 in a kubeadm built cluster running 1.18.8

• Kubernetes cluster kind:
  kubeadm

• Manifests:
  example.jsonnet

local kp =
  (import 'kube-prometheus/kube-prometheus.libsonnet') +
  (import 'kube-prometheus/kube-prometheus-kubeadm.libsonnet') +
  (import 'kube-prometheus/kube-prometheus-all-namespaces.libsonnet') +
  (import 'kube-prometheus/kube-prometheus-anti-affinity.libsonnet')
  {
    _config+:: {
      namespace: 'monitoring',
    },
  };

{ ['setup/0namespace-' + name]: kp.kubePrometheus[name] for name in std.objectFields(kp.kubePrometheus) } +
{
  ['setup/prometheus-operator-' + name]: kp.prometheusOperator[name]
  for name in std.filter((function(name) name != 'serviceMonitor'), std.objectFields(kp.prometheusOperator))
} +
// serviceMonitor is separated so that it can be created after the CRDs are ready
{ 'prometheus-operator-serviceMonitor': kp.prometheusOperator.serviceMonitor } +
{ ['node-exporter-' + name]: kp.nodeExporter[name] for name in std.objectFields(kp.nodeExporter) } +
{ ['kube-state-metrics-' + name]: kp.kubeStateMetrics[name] for name in std.objectFields(kp.kubeStateMetrics) } +
{ ['alertmanager-' + name]: kp.alertmanager[name] for name in std.objectFields(kp.alertmanager) } +
{ ['prometheus-' + name]: kp.prometheus[name] for name in std.objectFields(kp.prometheus) } +
{ ['prometheus-adapter-' + name]: kp.prometheusAdapter[name] for name in std.objectFields(kp.prometheusAdapter) } +
{ ['grafana-' + name]: kp.grafana[name] for name in std.objectFields(kp.grafana) }

kube-system services:

NAME                                           TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                        AGE     SELECTOR
kube-controller-manager-prometheus-discovery   ClusterIP   None             <none>        10257/TCP                      5h57m   component=kube-controller-manager
kube-dns                                       ClusterIP   10.96.0.10       <none>        53/UDP,53/TCP,9153/TCP         503d    k8s-app=kube-dns
kube-dns-prometheus-discovery                  ClusterIP   None             <none>        10055/TCP,10054/TCP            2d6h    k8s-app=kube-dns
kube-scheduler-prometheus-discovery            ClusterIP   None             <none>        10259/TCP                      5h57m   component=kube-scheduler
kubelet                                        ClusterIP   None             <none>        10250/TCP,10255/TCP,4194/TCP   491d    <none>
metrics-server                                 ClusterIP   10.103.192.119   <none>        443/TCP                        412d    k8s-app=metrics-server

[kehao95]

Hi, I had same issue before. But I was using GKE.
To my understanding, depending on the cluster setup. Some components are not available or visible to the monitoring system.
So I ended up disable the corresponding rules in config.jsonnet like this.

...
+ {
    # Prometheus patch
    prometheus +:: {
      # Adjust resource limits
      prometheus+:{
        spec+:{
          resources+:{
            requests+:{
              memory: "1Gi"
            },
          },
        }
      },
      # remove unwanted rule groups
      rules+:{
        spec+:{
          groups: [
          x for x in super.groups 
          if std.count([
            "kubernetes-system-controller-manager", # unavailable in GKE
            "kubernetes-system-scheduler",          # unavailable in GKE
            "kubernetes-system-apiserver",
            "general.rules"                         # don't use DeadMansSnitch for now
            ], x.name) == 0
          ],
        }
      }
    },
...

Hope this may help.

[brancz]

There is a jsonnet patch available to do this: https://github.com/prometheus-operator/kube-prometheus/blob/master/jsonnet/kube-prometheus/kube-prometheus-managed-cluster.libsonnet

[jeanluclariviere]

Thanks, it turns out this was related to a change in kubeadm which sets the bind-address to the local loopback if none is provided (kubernetes/kubernetes#86493). Setting a --bind-address for both corrects the issue, but this answer is also correct.

[balthazar]

Since that file was deleted in bdfa78a you can see it here for people wondering.

I'm using EKS and those alerts keep firing, not too sure on how to configure the flags mentioned in this comment. I'm not using the helm chart but the built manifests, and from what I gather, only the kubeProxy flag is available in the KubernetesControlPane interface.

Maybe @brancz can shed those light on this? Danke!

[balthazar]

It seems I have the incredible superpower to find the responses to my questions right after I post them.

The mentioned file is basically now an addon which you can simply enable this way:

local kp =                                                                      
    (import 'kube-prometheus/main.libsonnet') +                                   
    (import 'kube-prometheus/addons/managed-cluster.libsonnet') +