How to add custom labels to grafana dashboard configmaps? #606

mmn01-sky · 2020-07-09T11:01:04Z

mmn01-sky
Jul 9, 2020

Currently, I'm deploying prometheus-operator using the helm charts without the default grafana dashboards and prometheus alerts/rules. These are currently disabled.

I am now using the kube-prometheus repo to build and compile my alerts, rules and dashboards. The alerts and rules for prometheus worked great, i managed to modify them to include my custom labels for each alert.

However the grafana dashboards are proving problematic. Currently, my grafana is configured to automatically pull in any dashboards with a specific metadata label. This way, i don't need to constantly list all of my configmap dashboards into the grafana deployment as a volume mount.

I need to be able to add custom metadata labels to all of my grafana dashboard configmaps (grafana_datasource or custom_label):

apiVersion: v1
items:
- apiVersion: v1
  data:
    apiserver.json: |-
      { dashboard json here }
  kind: ConfigMap
  metadata:
    name: grafana-dashboard-apiserver
    namespace: monitoring
    grafana_datasource: true
    custom_label: custom_value

Below is my current libsonnet file:

local kp =
  (import 'kube-prometheus/kube-prometheus.libsonnet') +
  // Uncomment the following imports to enable its patches
  // (import 'kube-prometheus/kube-prometheus-anti-affinity.libsonnet') +
  // (import 'kube-prometheus/kube-prometheus-managed-cluster.libsonnet') +
  // (import 'kube-prometheus/kube-prometheus-node-ports.libsonnet') +
  // (import 'kube-prometheus/kube-prometheus-static-etcd.libsonnet') +
  // (import 'kube-prometheus/kube-prometheus-thanos-sidecar.libsonnet') +
  // (import 'kube-prometheus/kube-prometheus-custom-metrics.libsonnet') +
  {
    _config+:: {
      namespace: 'monitoring',
    },

    prometheusAlerts+:: {
      groups: std.map(
        function(group)
            group {
              rules: std.map(
                function(rule)
                  if std.objectHas(rule, 'labels') && std.objectHas(rule.labels, 'severity') && rule.labels.severity == "warning" then 
                      rule {
                        labels: {
                          application: "{{ if $labels.service }} {{ $labels.service }} {{ else }} UKISS GKE {{ end }}",
                          severity: "P4",
                          sparkGroup: "UK IS DOST Containerisation Support",
                          affectedCi: "{{ $labels.sparkCi }}",                      
                        },
                      }
                  else if std.objectHas(rule, 'labels') && std.objectHas(rule.labels, 'severity') && rule.labels.severity == "critical" then 
                      rule {
                        labels: {
                          application: "{{ if $labels.service }} {{ $labels.service }} {{ else }} UKISS GKE {{ end }}",
                          severity: "P3",
                          sparkGroup: "UK IS DOST Containerisation Support",
                          affectedCi: "{{ $labels.sparkCi }}",                         
                          },
                      }
                  else
                    rule,
                group.rules
              ),
            },
        super.groups
      ),
    },

    grafanaDashboards+:: {  //  monitoring-mixin compatibility
    'multi-cluster.json': (import 'grafana-dashboards/multi-cluster-dashboard.json'),
    },
    grafana+:: {
      dashboards+:: {  // use this method to import your dashboards to Grafana
        'multi-cluster.json': (import 'grafana-dashboards/multi-cluster-dashboard.json'),
      },
    },

  };

{ ['setup/0namespace-' + name]: kp.kubePrometheus[name] for name in std.objectFields(kp.kubePrometheus) } +
{
  ['setup/prometheus-operator-' + name]: kp.prometheusOperator[name]
  for name in std.filter((function(name) name != 'serviceMonitor'), std.objectFields(kp.prometheusOperator))
} +
// serviceMonitor is separated so that it can be created after the CRDs are ready
// { 'prometheus-operator-serviceMonitor': kp.prometheusOperator.serviceMonitor } +
// { ['node-exporter-' + name]: kp.nodeExporter[name] for name in std.objectFields(kp.nodeExporter) } +
// { ['kube-state-metrics-' + name]: kp.kubeStateMetrics[name] for name in std.objectFields(kp.kubeStateMetrics) } +
// { ['alertmanager-' + name]: kp.alertmanager[name] for name in std.objectFields(kp.alertmanager) } +
{ ['prometheus-' + name]: kp.prometheus[name] for name in std.objectFields(kp.prometheus) } +
// { ['prometheus-adapter-' + name]: kp.prometheusAdapter[name] for name in std.objectFields(kp.prometheusAdapter) } +
{ ['grafana-' + name]: kp.grafana[name] for name in std.objectFields(kp.grafana) }

It doesn't seem like there's a possible way to do this.

Anyone have any ideas?

Answered by brancz

Jul 9, 2020

My apologies, I think I missed the items array:

local kp =
  (import 'kube-prometheus/kube-prometheus.libsonnet') +
  // Uncomment the following imports to enable its patches
  // (import 'kube-prometheus/kube-prometheus-anti-affinity.libsonnet') +
  // (import 'kube-prometheus/kube-prometheus-managed-cluster.libsonnet') +
  // (import 'kube-prometheus/kube-prometheus-node-ports.libsonnet') +
  // (import 'kube-prometheus/kube-prometheus-static-etcd.libsonnet') +
  // (import 'kube-prometheus/kube-prometheus-thanos-sidecar.libsonnet') +
  // (import 'kube-prometheus/kube-prometheus-custom-metrics.libsonnet') +
  {
    _config+:: {
      namespace: 'monitoring',
    },

    prometheusAlert…

View full answer

brancz · 2020-07-09T13:48:10Z

brancz
Jul 9, 2020
Maintainer

without having tested this, something like this should work (see the dashboardDefinitions sibling next to the in your snippet existing dashboards)

local kp =
  (import 'kube-prometheus/kube-prometheus.libsonnet') +
  // Uncomment the following imports to enable its patches
  // (import 'kube-prometheus/kube-prometheus-anti-affinity.libsonnet') +
  // (import 'kube-prometheus/kube-prometheus-managed-cluster.libsonnet') +
  // (import 'kube-prometheus/kube-prometheus-node-ports.libsonnet') +
  // (import 'kube-prometheus/kube-prometheus-static-etcd.libsonnet') +
  // (import 'kube-prometheus/kube-prometheus-thanos-sidecar.libsonnet') +
  // (import 'kube-prometheus/kube-prometheus-custom-metrics.libsonnet') +
  {
    _config+:: {
      namespace: 'monitoring',
    },

    prometheusAlerts+:: {
      groups: std.map(
        function(group)
            group {
              rules: std.map(
                function(rule)
                  if std.objectHas(rule, 'labels') && std.objectHas(rule.labels, 'severity') && rule.labels.severity == "warning" then 
                      rule {
                        labels: {
                          application: "{{ if $labels.service }} {{ $labels.service }} {{ else }} UKISS GKE {{ end }}",
                          severity: "P4",
                          sparkGroup: "UK IS DOST Containerisation Support",
                          affectedCi: "{{ $labels.sparkCi }}",                      
                        },
                      }
                  else if std.objectHas(rule, 'labels') && std.objectHas(rule.labels, 'severity') && rule.labels.severity == "critical" then 
                      rule {
                        labels: {
                          application: "{{ if $labels.service }} {{ $labels.service }} {{ else }} UKISS GKE {{ end }}",
                          severity: "P3",
                          sparkGroup: "UK IS DOST Containerisation Support",
                          affectedCi: "{{ $labels.sparkCi }}",                         
                          },
                      }
                  else
                    rule,
                group.rules
              ),
            },
        super.groups
      ),
    },

    grafanaDashboards+:: {  //  monitoring-mixin compatibility
    'multi-cluster.json': (import 'grafana-dashboards/multi-cluster-dashboard.json'),
    },
    grafana+:: {
      dashboards+:: {  // use this method to import your dashboards to Grafana
        'multi-cluster.json': (import 'grafana-dashboards/multi-cluster-dashboard.json'),
      },
      dashboardDefinitions: std.map(
        function(cm) cm + {metadata+:{labels+: {label: value}}},
      super.dashboardDefinitions),
    },

  };

{ ['setup/0namespace-' + name]: kp.kubePrometheus[name] for name in std.objectFields(kp.kubePrometheus) } +
{
  ['setup/prometheus-operator-' + name]: kp.prometheusOperator[name]
  for name in std.filter((function(name) name != 'serviceMonitor'), std.objectFields(kp.prometheusOperator))
} +
// serviceMonitor is separated so that it can be created after the CRDs are ready
// { 'prometheus-operator-serviceMonitor': kp.prometheusOperator.serviceMonitor } +
// { ['node-exporter-' + name]: kp.nodeExporter[name] for name in std.objectFields(kp.nodeExporter) } +
// { ['kube-state-metrics-' + name]: kp.kubeStateMetrics[name] for name in std.objectFields(kp.kubeStateMetrics) } +
// { ['alertmanager-' + name]: kp.alertmanager[name] for name in std.objectFields(kp.alertmanager) } +
{ ['prometheus-' + name]: kp.prometheus[name] for name in std.objectFields(kp.prometheus) } +
// { ['prometheus-adapter-' + name]: kp.prometheusAdapter[name] for name in std.objectFields(kp.prometheusAdapter) } +
{ ['grafana-' + name]: kp.grafana[name] for name in std.objectFields(kp.grafana) }

1 reply

mmn01-sky Jul 9, 2020
Author

Thanks for your response.

I added the following to my libsonnet file

    grafana+:: {
      dashboards+:: {  // use this method to import your dashboards to Grafana
        'multi-cluster.json': (import 'grafana-dashboards/multi-cluster-dashboard.json'),

        dashboardDefinitions: std.map(
          function(cm) cm + {metadata+:{labels+: {grafana_datasource: 'true'}}},
        super.dashboardDefinitions),
      },
    },

I don't get any errors but I dont see my new label "grafana_datasource" created in the new configmaps created for the grafana dashboards.

Output after running script:

./build.sh ukiss-alerts.libsonnet 
+ set -o pipefail
++ pwd
+ PATH=/Users/mmn01/git/apps/kube-prometheus/tmp/bin:/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
+ SPARK_GROUP='UK IS DOST Containerisation Support'
+ rm -rf manifests
+ mkdir -p manifests/setup
+ jsonnet -J vendor -m manifests ukiss-alerts.libsonnet --ext-str 'SPARK_GROUP=UK IS DOST Containerisation Support'
+ xargs '-I{}' sh -c 'cat {} | gojsontoyaml > {}.yaml' -- '{}'
+ find manifests -type f '!' -name '*.yaml' -delete
+ rm -f kustomization

brancz · 2020-07-09T14:42:53Z

brancz
Jul 9, 2020
Maintainer

My apologies, I think I missed the items array:

local kp =
  (import 'kube-prometheus/kube-prometheus.libsonnet') +
  // Uncomment the following imports to enable its patches
  // (import 'kube-prometheus/kube-prometheus-anti-affinity.libsonnet') +
  // (import 'kube-prometheus/kube-prometheus-managed-cluster.libsonnet') +
  // (import 'kube-prometheus/kube-prometheus-node-ports.libsonnet') +
  // (import 'kube-prometheus/kube-prometheus-static-etcd.libsonnet') +
  // (import 'kube-prometheus/kube-prometheus-thanos-sidecar.libsonnet') +
  // (import 'kube-prometheus/kube-prometheus-custom-metrics.libsonnet') +
  {
    _config+:: {
      namespace: 'monitoring',
    },

    prometheusAlerts+:: {
      groups: std.map(
        function(group)
            group {
              rules: std.map(
                function(rule)
                  if std.objectHas(rule, 'labels') && std.objectHas(rule.labels, 'severity') && rule.labels.severity == "warning" then 
                      rule {
                        labels: {
                          application: "{{ if $labels.service }} {{ $labels.service }} {{ else }} UKISS GKE {{ end }}",
                          severity: "P4",
                          sparkGroup: "UK IS DOST Containerisation Support",
                          affectedCi: "{{ $labels.sparkCi }}",                      
                        },
                      }
                  else if std.objectHas(rule, 'labels') && std.objectHas(rule.labels, 'severity') && rule.labels.severity == "critical" then 
                      rule {
                        labels: {
                          application: "{{ if $labels.service }} {{ $labels.service }} {{ else }} UKISS GKE {{ end }}",
                          severity: "P3",
                          sparkGroup: "UK IS DOST Containerisation Support",
                          affectedCi: "{{ $labels.sparkCi }}",                         
                          },
                      }
                  else
                    rule,
                group.rules
              ),
            },
        super.groups
      ),
    },

    grafanaDashboards+:: {  //  monitoring-mixin compatibility
    'multi-cluster.json': (import 'grafana-dashboards/multi-cluster-dashboard.json'),
    },
    grafana+:: {
      dashboards+:: {  // use this method to import your dashboards to Grafana
        'multi-cluster.json': (import 'grafana-dashboards/multi-cluster-dashboard.json'),
      },
      dashboardDefinitions+: {items: std.map(
        function(cm) cm + {metadata+:{labels+: {label: value}}},
      super.items)},
    },

  };

{ ['setup/0namespace-' + name]: kp.kubePrometheus[name] for name in std.objectFields(kp.kubePrometheus) } +
{
  ['setup/prometheus-operator-' + name]: kp.prometheusOperator[name]
  for name in std.filter((function(name) name != 'serviceMonitor'), std.objectFields(kp.prometheusOperator))
} +
// serviceMonitor is separated so that it can be created after the CRDs are ready
// { 'prometheus-operator-serviceMonitor': kp.prometheusOperator.serviceMonitor } +
// { ['node-exporter-' + name]: kp.nodeExporter[name] for name in std.objectFields(kp.nodeExporter) } +
// { ['kube-state-metrics-' + name]: kp.kubeStateMetrics[name] for name in std.objectFields(kp.kubeStateMetrics) } +
// { ['alertmanager-' + name]: kp.alertmanager[name] for name in std.objectFields(kp.alertmanager) } +
{ ['prometheus-' + name]: kp.prometheus[name] for name in std.objectFields(kp.prometheus) } +
// { ['prometheus-adapter-' + name]: kp.prometheusAdapter[name] for name in std.objectFields(kp.prometheusAdapter) } +
{ ['grafana-' + name]: kp.grafana[name] for name in std.objectFields(kp.grafana) }

3 replies

mmn01-sky Jul 9, 2020
Author

That worked, thank you very much for your help! I do have another question related to jsonnet but i'll create a new discussion.

Thanks again!

dushyy Nov 1, 2022

Hello,

The same example doesn't seem to work for me. I'm trying to add ingress-nginx-mixin to kube-prometheus and for grafana-sc-dashboard sidecar to pickup this dashboard it needs the label grafana_dashboard set

My libsonnet file

local addMixin = (import 'kube-prometheus/lib/mixin.libsonnet');

//ingressNginx mixin
local ingressNginxMixin = addMixin({
  name: 'ingressNginx',
  mixin: (import 'ingress-nginx-mixin/mixin.libsonnet') + {
    _config+: {},  // mixin configuration object
  },
});


local kp =
  (import 'kube-prometheus/main.libsonnet') +
  {
    values+:: {
      common+: {
        namespace: 'monitoring',
      },
      grafana+: {
        // Adding new dashboard to grafana. This will create new dashboard configMap
        dashboards+: ingressNginxMixin.grafanaDashboards,

        //add grafana_dashboard: "1" label for grafana-sc-dashboard sidecar to pickup new dashboards
        dashboardDefinitions+: {items: std.map(
          function(cm) cm + {metadata+:{labels+: {grafana_dashboard: '1'}}},
        super.items)},
      },
   },
  };


//we dont need the below as the default CRDs, dashboards and rules are deployed via kube-prometheus-stack

//{ ['00namespace-' + name]: kp.kubePrometheus[name] for name in std.objectFields(kp.kubePrometheus) } +
//{ ['0prometheus-operator-' + name]: kp.prometheusOperator[name] for name in std.objectFields(kp.prometheusOperator) } +
//{ ['node-exporter-' + name]: kp.nodeExporter[name] for name in std.objectFields(kp.nodeExporter) } +
//{ ['kube-state-metrics-' + name]: kp.kubeStateMetrics[name] for name in std.objectFields(kp.kubeStateMetrics) } +
//{ ['alertmanager-' + name]: kp.alertmanager[name] for name in std.objectFields(kp.alertmanager) } +
//{ ['prometheus-' + name]: kp.prometheus[name] for name in std.objectFields(kp.prometheus) } +
// Rendering prometheusRules object. This is an object compatible with prometheus-operator CRD definition for prometheusRule

// generates dashboards as configmap - generates all default dashboards too
{ ['grafana-' + name]: kp.grafana[name] for name in std.objectFields(kp.grafana) } +

//non default mixins inclusions below
{ 'external-mixins/ingress-nginx-mixin': ingressNginxMixin.prometheusRules }

Build runs fine and i can see the dashboard and rules yamls being generated

$ ./build.sh custom-mixins.jsonnet
+ set -o pipefail
++ pwd
+ PATH=/home/pb-jb-ubuntu/work/k8s-mixins-common/tmp/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/pb-jb-ubuntu/go/bin
+ rm -rf manifests
+ mkdir -p manifests/setup
+ mkdir -p manifests/external-mixins
+ jsonnet -J vendor -m manifests custom-mixins.jsonnet
+ xargs '-I{}' sh -c 'cat {} | gojsontoyaml > {}.yaml' -- '{}'
+ find manifests -type f '!' -name '*.yaml' -delete
+ rm -f kustomization

However, The dashboard configMaps in manifests/grafana-dashboardDefinitions.yaml don't have the new label.

...
kind: ConfigMap
metadata:
  labels:
    app.kubernetes.io/component: grafana
    app.kubernetes.io/name: grafana
    app.kubernetes.io/part-of: kube-prometheus
    app.kubernetes.io/version: 9.1.7
  name: grafana-dashboard-alertmanager-overview
  namespace: monitoring

I looked at grafana.libsonnet code and i suspect it also needs some changes for this work ?

Any guidance will be really helpful.

Thanks
Dushyanth

dushyy Nov 2, 2022

I was doing it wrong - i nested everything under the values block. Fixed that and i can see appropriate labels now.

BTW - shouldn't grafana_dashboards: "1" be a default label now for all dashboards generated declaratively ?

working libsonnet to add external mixins with kube-prometheus

local addMixin = (import 'kube-prometheus/lib/mixin.libsonnet');

//ingressNginx mixin
local ingressNginxMixin = addMixin({
  name: 'ingress-nginx',
  namespace: 'monitoring',
  mixin: (import 'ingress-nginx-mixin/mixin.libsonnet') + {
    _config+: {},  // mixin configuration object
  },
});


local kp =
  (import 'kube-prometheus/main.libsonnet') +
  {
    values+:: {
      common+: {
        namespace: 'monitoring',
      },

      grafana+: {
        // Adding new dashboard to grafana. This will create new dashboard configMap
        dashboards+: ingressNginxMixin.grafanaDashboards,
      },
    },

    _config+:: {
      namespace: 'monitoring',
    },

    grafana+: {
      //add grafana_dashboard: "1" label for grafana-sc-dashboard sidecar to pickup new dashboards
      dashboardDefinitions+: { items: std.map(
        function(cm) cm { metadata+: { labels+: { grafana_dashboard: '1' } } },
        super.items
      ) },
    },
  };


//we dont need the below as the default CRDs, dashboards and rules are deployed via kube-prometheus-stack

//{ ['00namespace-' + name]: kp.kubePrometheus[name] for name in std.objectFields(kp.kubePrometheus) } +
//{ ['0prometheus-operator-' + name]: kp.prometheusOperator[name] for name in std.objectFields(kp.prometheusOperator) } +
//{ ['node-exporter-' + name]: kp.nodeExporter[name] for name in std.objectFields(kp.nodeExporter) } +
//{ ['kube-state-metrics-' + name]: kp.kubeStateMetrics[name] for name in std.objectFields(kp.kubeStateMetrics) } +
//{ ['alertmanager-' + name]: kp.alertmanager[name] for name in std.objectFields(kp.alertmanager) } +
//{ ['prometheus-' + name]: kp.prometheus[name] for name in std.objectFields(kp.prometheus) } +
// Rendering prometheusRules object. This is an object compatible with prometheus-operator CRD definition for prometheusRule

// this generates dashboards as configmap - generates grafana deployment specs and all default dashboards too
// `kubectl apply -f manifests/` will update all changes  if any including grafana deployment 
// for selective apply of dashboard & rules: `kubectl apply -f manifests/grafana-dashboardDefinitions.yaml -f manifests/external-mixins/*.yaml`
{ ['grafana-' + name]: kp.grafana[name] for name in std.objectFields(kp.grafana) } +

//non default mixins inclusions below
{ 'external-mixins/ingress-nginx-mixin': ingressNginxMixin.prometheusRules }

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

How to add custom labels to grafana dashboard configmaps? #606

{{title}}

Replies: 2 comments 4 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

How to add custom labels to grafana dashboard configmaps? #606

mmn01-sky Jul 9, 2020

Replies: 2 comments · 4 replies

brancz Jul 9, 2020 Maintainer

mmn01-sky Jul 9, 2020 Author

brancz Jul 9, 2020 Maintainer

mmn01-sky Jul 9, 2020 Author

dushyy Nov 1, 2022

My libsonnet file

dushyy Nov 2, 2022

working libsonnet to add external mixins with kube-prometheus

mmn01-sky
Jul 9, 2020

Replies: 2 comments 4 replies

brancz
Jul 9, 2020
Maintainer

mmn01-sky Jul 9, 2020
Author

brancz
Jul 9, 2020
Maintainer

mmn01-sky Jul 9, 2020
Author