Strimzi prometheus server pod refuses to deploy

[davehouser1]

I am following the instructions found here
Also I found an online blog with exact commands used found here

I am able to deploy everything up to kubectl -n kafka apply -f prometheus.yaml. After doing so, kubernetes shows

clusterrole.rbac.authorization.k8s.io/prometheus-server created
serviceaccount/prometheus-server created
clusterrolebinding.rbac.authorization.k8s.io/prometheus-server created
prometheus.monitoring.coreos.com/prometheus created

However, nothing happens. no logs, no events, no pods, no services are created. I see nothing. I am using the same files the blog poster uses which are the same exact files that are in the Strimzi repo. I thought his maybe because of load balacning or some other rule I configured on my custom deployment, therefore I tried deploying my zk/kafka cluster with just a vanilla ephemeral deployment. Still nothing.
Why does a prometheus-server pod never get deployed?

Here is the manifest I am using for promethus.yaml

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: prometheus-server
  labels:
    app: strimzi
rules:
  - apiGroups: [""]
    resources:
      - nodes
      - nodes/proxy
      - services
      - endpoints
      - pods
    verbs: ["get", "list", "watch"]
  - apiGroups:
      - extensions
    resources:
      - ingresses
    verbs: ["get", "list", "watch"]
  - nonResourceURLs: ["/metrics"]
    verbs: ["get"]

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: prometheus-server
  labels:
    app: strimzi

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: prometheus-server
  labels:
    app: strimzi
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: prometheus-server
subjects:
  - kind: ServiceAccount
    name: prometheus-server
    namespace: kafka


---
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: prometheus
  labels:
    app: strimzi
spec:
  replicas: 1
  serviceAccountName: prometheus-server
  podMonitorSelector:
    matchLabels:
      app: strimzi
  resources:
    requests:
      memory: 400Mi
  enableAdminAPI: false
  ruleSelector:
    matchLabels:
      role: alert-rules
      app: strimzi
  alerting:
    alertmanagers:
    - namespace: kafka
      name: alertmanager
      port: alertmanager
  additionalScrapeConfigs:
    name: additional-scrape-configs
    key: prometheus-additional.yaml

[scholzj]

Did you installed the Prometheus Operator first?

[scholzj]

That is the chapter before the one you linked: https://strimzi.io/docs/operators/latest/full/deploying.html#proc-metrics-deploying-prometheus-operator-str

[davehouser1]

Yes sir, I did, and it shows deployed and running.

…

On Mon, May 10, 2021, 10:46 AM Jakub Scholz ***@***.***> wrote: Did you installed the Prometheus Operator first? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#4937 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AOL4D3KFDGGAFAWHF5BC4RTTM7WVBANCNFSM44RMCUFA> .

[scholzj]

So when you create the kind: Prometheus resource, that should create the Prometheus cluster. If it did not, you probably need to check the operator logs / operator deployment (e.g. does it watch the right namespace etc.)

[davehouser1]

ah ha, logs I needed to look at were in the prometheus-operator. I see the error thats happening over and over now.

evel=info ts=2021-05-10T15:05:30.783420845Z caller=operator.go:1163 component=prometheusoperator msg="sync prometheus" key=kafka/prometheus
level=error ts=2021-05-10T15:05:30.819751772Z caller=klog.go:96 component=k8s_client_runtime func=ErrorDepth msg="Sync \"kafka/prometheus\" failed: creating config failed: generating config failed: unmarshalling additional scrape configs failed: yaml: unmarshal errors:\n  line 1: cannot unmarshal !!map into []yaml.MapSlice"

Any recommendations on how to resolve?

[scholzj]

I have only little idea how the scrape configs should look like. Maybe @sknot-rh or @ppatierno can help with this?

[davehouser1]

Got it working. Few things I was doing wrong:

1. I missread this section of the instructions

Edit the additionalScrapeConfigs property in the prometheus.yaml file to include the name of the Secret and the prometheus-additional.yaml file.

I adjusted the last line of the prometheus.yaml file for "key: prometheus-additional.yaml" to the renamed prometheus-additional.yaml file I created. This breaks the deployment. So I put it back to the default to what it originally said.

2. I created my secret but never APPLIED the actual promethus-additinal.yaml file, needed to do that.

Thank you again for your quick and helpful replies @scholzj, most appreciated as always.
I may have more questions about the prometheus / graphana deployment but I will open a new string if that happens.

[sknot-rh]

It seems your additional scrape configs are misconfigured. Can you doublecheck&provide them to verify?