Scraping Kafka and Zookeeper metrics ver 20 in Prometheus

[athiramanohar]

Hi,

I am trying to scrape kafka and zookeeepr metrics for version 20 in prometheus. I have added Pod Monitors in prometheus for cluster-operator, entity-operator and kafka-resource metrics as per https://github.com/strimzi/strimzi-kafka-operator/blob/release-0.20.x/examples/metrics/prometheus-install/strimzi-pod-monitor.yaml. Kafka is deployed using https://github.com/strimzi/strimzi-kafka-operator/blob/release-0.20.x/examples/metrics/kafka-metrics.yaml. But we are not seeing any metrics for kafka and zookeeper in Prometheus.
Available metrics are for strimzi operator pod, entity operator and kafka exporter.

I also tried adding a kafka pod monitor and service monitor with below configurations, still unable to read metrics.

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: kafka-metrics
  labels:
    app: strimzi
spec:
  selector:
    matchLabels:
      strimzi.io/kind: Kafka
  namespaceSelector:
    matchNames:
    - kafka-ver-20-sal
  podMetricsEndpoints:
  - path: /metrics
    port: plain

apiVersion: monitoring.coreos.com/v1
  kind: ServiceMonitor
  metadata:
    name: kafka-service-monitor
    labels:
      app: strimzi
  spec:
    selector:
      matchLabels:
        strimzi.io/kind: Kafka
    namespaceSelector:
      matchNames:
        - <ns>       
    endpoints:
    - port: prometheus
      honorLabels: true
      interval: 10s
      scrapeTimeout: 10s
      path: /metrics
      scheme: http
      relabelings:
      - sourceLabels: [__meta_kubernetes_endpoints_name]
        separator: ;
        regex: prometheus-kube-state-metrics
        replacement: $1
        action: keep
      - separator: ;
        regex: __meta_kubernetes_service_label_(.+)
        replacement: $1
        action: labelmap
      - sourceLabels: [__meta_kubernetes_namespace]
        separator: ;
        regex: (.*)
        targetLabel: namespace
        replacement: $1
        action: replace
      - sourceLabels: [__meta_kubernetes_namespace]
        separator: ;
        regex: (.*)
        targetLabel: kubernetes_namespace
        replacement: $1
        action: replace
      - sourceLabels: [__meta_kubernetes_service_name]
        separator: ;
        regex: (.*)
        targetLabel: kubernetes_name
        replacement: $1
        action: replace
      - sourceLabels: [__meta_kubernetes_pod_node_name]
        separator: ;
        regex: (.*)
        targetLabel: node_name
        replacement: $1
        action: replace
      - sourceLabels: [__meta_kubernetes_pod_host_ip]
        separator: ;
        regex: (.*)
        targetLabel: node_ip
        replacement: $1
        action: replace
    - port: prometheus
      honorLabels: true
      interval: 10s
      scrapeTimeout: 10s
      path: /metrics
      scheme: http
      relabelings:
      - sourceLabels: [__meta_kubernetes_endpoints_name]
        separator: ;
        regex: prometheus-node-exporter
        replacement: $1
        action: keep
      - separator: ;
        regex: __meta_kubernetes_service_label_(.+)
        replacement: $1
        action: labelmap
      - sourceLabels: [__meta_kubernetes_namespace]
        separator: ;
        regex: (.*)
        targetLabel: namespace
        replacement: $1
        action: replace
      - sourceLabels: [__meta_kubernetes_namespace]
        separator: ;
        regex: (.*)
        targetLabel: kubernetes_namespace
        replacement: $1
        action: replace
      - sourceLabels: [__meta_kubernetes_service_name]
        separator: ;
        regex: (.*)
        targetLabel: kubernetes_name
        replacement: $1
        action: replace
      - sourceLabels: [__meta_kubernetes_pod_node_name]
        separator: ;
        regex: (.*)
        targetLabel: node_name
        replacement: $1
        action: replace
      - sourceLabels: [__meta_kubernetes_pod_host_ip]
        separator: ;
        regex: (.*)
        targetLabel: node_ip
        replacement: $1
        action: replace

[scholzj]

As shown in this file: https://github.com/strimzi/strimzi-kafka-operator/blob/release-0.20.x/examples/metrics/prometheus-install/strimzi-pod-monitor.yaml ... you should use the PodMonitor to collect the metrics. You example above seems to have ServiceMonitor.

[athiramanohar]

Hi scholzj,
I have added podMonitor as per https://github.com/strimzi/strimzi-kafka-operator/blob/release-0.20.x/examples/metrics/prometheus-install/strimzi-pod-monitor.yaml. Since I was not getting metrics, I added servicemonitor as well, but that also didnt help in getting kafka and zookeeper pod metrics.

[scholzj]

So if you used the PodMonitors from the examples ... do you see the targets in Prometheus?

[athiramanohar]

Yes I can see targets in Prometheus, as shown in the image in the issue above, but there are no metrics listed.

Targets are available for Kafka podMonitor as well as Kafka service monitor. But no metrics are scraped from kafka pods or service.

Where are metrics are coming from PodMonitors deployed for strimzi-operator, entity-operator and exporter.

[scholzj]

The kafka-resource-metrics PodMonitor should scrape metrics from Kafka:

strimzi-kafka-operator/examples/metrics/prometheus-install/strimzi-pod-monitor.yaml

Line 55 in 5e5e740

name: kafka-resources-metrics

... it shows 2 targets ... so what happens when you click show more? Also, can you share your Kafka CR? Did you enabled the metrics there?

[anragen]

@scholzj Can you help me? I installed prometheus-operator-9.3.1 (by helm2) by default values. When i add podmonitor i see nothing in targets. When i create me own servicemonitor i see the new target.

[scholzj]

@anragen Are you working with @athiramanohar? If not it might be best to start separate discussion to avoid mixing up the issues: https://github.com/strimzi/strimzi-kafka-operator/discussions What is the version of your Prometheus Operator? Does it support PodMonitors?

[anragen]

@scholzj no i a not with @athiramanohar. I solved the problem. I set labels release.

[athiramanohar]

Hi Scholzj,
The kafka-resource-metrics PodMonitor is showing metrics from kafka-exporter pod. The endpoint is as shown below:

So do you mean these are the only Kafka metrics available? We used to get more kafka and zookeeper related metrics when using kafka 17 version, from exporter service on port 9404. But many of those metrics are not available now in 20 version. So I wanted to know if we need to deploy separate PodMonitors or servicemonitors for kafka and zookeeper?

[scholzj]

The exporter shares only the lag related metrics - these are addition to the other metrics. It does not provide any Kafka or Zookeeper metrics.

From 0.20, you should use PodMonitors for everything since the Prometheus ports are not exposed on the services anymore. So if you have ServiceMonitors, you should ideally change them to PodMonitors (as in the examples). Alternatively you can keep using ServiceMonitor and just add some custom services with the 9404 port ... but the first option is normally easier.

[athiramanohar]

Ok. I tried a separate PodMonitor for Kafka, but endpoints are not available under the target as shown the screenshot below. Kindly let me know if the PodMonitor configuration needs any more steps to be added.

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: kafka-metrics
  labels:
    app: strimzi
spec:
  selector:
    matchLabels:
      strimzi.io/kind: Kafka
  namespaceSelector:
    matchNames:
    - <namespace>
  podMetricsEndpoints:
  - path: /metrics
    port: plain`

[scholzj]

I think you have a wrong port there? The metrics are IMHO on port tcp-prometheus or something. You should check what is used in the PodMonitors provided by Strimzi

[athiramanohar]

Hi Scholjz, I modified the port to tcp-prometheus, and observed that the endpoint is same as the kafka-exporter that I received from Kafka-resource podmonitor added, whose endpoint is already on this port. Only kafka metrics I am receiving are:

1. kafka_brokers{}
2. kafka_exporter_build_info{}

Whereas while using 0.17 version, I used to receive metrics such as kafka_server*, kafka_network*, kafka_controller*, kafka_security*, kafka_log*, kafka_topic* etc, which were from services like bootstrap, zookeeper client and exporter. So I would like to know how I can get all these metrics in 0.20 version by using PodMonitors.

[scholzj]

As I said, since 0.20, the services do not expose the metrics port. So you need to install the PodMonitors from the examples. Prometheus normally scrapes the metrics after you create them. But I'm afraid I have no clue why they don' work for you. Plus you seem to be using different settings than what is in our examples - so it is hard to understand what do you have or don't have really running there.

[sknot-rh]

@athiramanohar did you manage to get it working?

[athiramanohar]

Hi @sknot-rh , not yet. I am still not able to get the desired metrics.

[athiramanohar]

@scholzj I am using the configurations as per https://github.com/strimzi/strimzi-kafka-operator/blob/release-0.20.x/examples/metrics/prometheus-install/strimzi-pod-monitor.yaml. Kafka is deployed using https://github.com/strimzi/strimzi-kafka-operator/blob/release-0.20.x/examples/metrics/kafka-metrics.yaml.

[scholzj]

But the PodMonitors above are different from what you are sharing as YAML. For example there is nothing like

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: kafka-metrics
  labels:
    app: strimzi
spec:
  selector:
    matchLabels:
      strimzi.io/kind: Kafka
  namespaceSelector:
    matchNames:
    - <namespace>
  podMetricsEndpoints:
  - path: /metrics
    port: plain

there.

[athiramanohar]

Yeah this one I added extra, as I was not able to get kafka-metrics, I have added the podMonitors as per the official page as well.

[athiramanohar]

Hi, I am able to get all required metrics when my kafka and prometheus cluster are in the same namespace. But if prometheus is in a different namespace, kafka and zookeeper metrics are not scraped. Is it the expected behavior or can we get the metrics even if prometheus is deployed in a separate namespace in the same GKE cluster?

[scholzj]

I run Prometheus in different namespace as well in my home cluster. I create the PodMonitor in the Prometheus operator namespace and they have inside the specification of the namespace where the pods should be monitored. You also need to make sure you don't have any network policies to prevent the access (such as locked down access between namespaces etc.).

[athiramanohar]

Ok, I just deleted the networkpolicies in the namespace, still I am not getting the kafka and zookeeper metrics