ServiceMonitor not added

[urbaman]

Hi,

I have a service/servicemonitor not added to prometheus.

kubectl -n monitoring get secret prometheus-kube-prometheus-stack-prometheus -ojson | jq -r '.data["prometheus.yaml.gz"]' | base64 -d | gunzip | grep cert-manager

Does not give any result, but the servicemonitor is seeing the service

kubectl get services -l "$(kubectl get servicemonitors -n "cert-manager" "cert-manager" -o template='{{ $first := 1 }}{{ range $key, $value := .spec.selector.matchLabels }}{{ if eq $first 0 }},{{end}}{{ $key }}={{ $value }}{{ $first = 0 }}{{end}}')" -n cert-manager
NAME           TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
cert-manager   ClusterIP   172.16.181.11   <none>        9402/TCP   108m

And the service is showing the metrics.

This is the service/servicemonitor:

apiVersion: v1
kind: Service
metadata:
  annotations:
    meta.helm.sh/release-name: cert-manager
    meta.helm.sh/release-namespace: cert-manager
  creationTimestamp: "2023-08-30T14:31:25Z"
  labels:
    app: cert-manager
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: cert-manager
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: cert-manager
    app.kubernetes.io/version: v1.12.3
    helm.sh/chart: cert-manager-v1.12.3
  name: cert-manager
  namespace: cert-manager
  resourceVersion: "31665009"
  uid: 1037f805-62e4-451f-bf26-97550d5ab02d
spec:
  clusterIP: 172.16.181.11
  clusterIPs:
  - 172.16.181.11
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - name: tcp-prometheus-servicemonitor
    port: 9402
    protocol: TCP
    targetPort: 9402
  selector:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: cert-manager
    app.kubernetes.io/name: cert-manager
  sessionAffinity: None
  type: ClusterIP
status:
  loadBalancer: {}
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  annotations:
    meta.helm.sh/release-name: cert-manager
    meta.helm.sh/release-namespace: cert-manager
  creationTimestamp: "2023-08-30T14:31:26Z"
  generation: 1
  labels:
    app: cert-manager
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: cert-manager
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: cert-manager
    app.kubernetes.io/version: v1.12.3
    helm.sh/chart: cert-manager-v1.12.3
    release: kube-prometheus-stack
  name: cert-manager
  namespace: cert-manager
  resourceVersion: "31669765"
  uid: 4b9a21ae-7805-40ae-bbf0-b72157ba39eb
spec:
  endpoints:
  - honorLabels: false
    interval: 60s
    path: /metrics
    scrapeTimeout: 30s
    targetPort: 9402
  jobLabel: cert-manager
  selector:
    matchLabels:
      app.kubernetes.io/component: controller
      app.kubernetes.io/instance: cert-manager
      app.kubernetes.io/name: cert-manager

The following service/servicemonitor is properly added and scraped:

apiVersion: v1
kind: Service
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"labels":{"k8s-app":"calico-node"},"name":"felix-metrics-svc","namespace":"calico-system"},"spec":{"clusterIP":"None","ports":[{"name":"metrics-port","port":9091,"targetPort":9091}],"selector":{"k8s-app":"calico-node"}}}
  creationTimestamp: "2023-08-30T13:14:36Z"
  labels:
    k8s-app: calico-node
  name: felix-metrics-svc
  namespace: calico-system
  resourceVersion: "31648704"
  uid: 2524f766-0468-458a-a7c9-eb1131544e82
spec:
  clusterIP: None
  clusterIPs:
  - None
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - name: metrics-port
    port: 9091
    protocol: TCP
    targetPort: 9091
  selector:
    k8s-app: calico-node
  sessionAffinity: None
  type: ClusterIP
status:
  loadBalancer: {}
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"monitoring.coreos.com/v1","kind":"ServiceMonitor","metadata":{"annotations":{},"labels":{"k8s-app":"calico-node","release":"kube-prometheus-stack"},"name":"felix-metrics-svc-sm","namespace":"calico-system"},"spec":{"endpoints":[{"interval":"15s","path":"/metrics","port":"metrics-port"}],"jobLabel":"felix-metrics","namespaceSelector":{"matchNames":["calico-system"]},"selector":{"matchLabels":{"k8s-app":"calico-node"}}}}
  creationTimestamp: "2023-08-30T13:14:38Z"
  generation: 1
  labels:
    k8s-app: calico-node
    release: kube-prometheus-stack
  name: felix-metrics-svc-sm
  namespace: calico-system
  resourceVersion: "31648105"
  uid: 663d568a-651e-4d86-aebc-b373536d42c7
spec:
  endpoints:
  - interval: 15s
    path: /metrics
    port: metrics-port
  jobLabel: felix-metrics
  namespaceSelector:
    matchNames:
    - calico-system
  selector:
    matchLabels:
      k8s-app: calico-node

What's wrong with the cert-manager servicemonitor?

[urbaman]

This is what happens in the operator logs:

msg="get namespace to enqueue Prometheus instances failed: namespace \"cert-manager\" does not exist"