Adding a custom prometheusrules in addition prometheus-k8s-rules

[rupadhya1]

The default prometheusrules that gets installed is the prometheus-k8s-rules.
If I were to add another custom-rules, what would be the syntax in jsonnet?

I have looked at the docs but they all talk about updating existing groups or adding groups to the existing rule.

Is there a way to add new custom prometheusrules with a different name (Ex: custom-k8s-rules in addition to prometheus-k8s-rules)?

[brancz]

Yes you just create entirely additional/separate PrometheusRule objects. You don't even necessarily need to do that in jsonnet, but in jsonnet you would just define another object something along the lines of:

{
  apiVersion: 'monitoring.coreos.com/v1',
  kind: 'PrometheusRule',
  metadata: {
    labels: {
      prometheus: 'my-prometheus', //  // keep in mind that this namespace must be selected by your `ruleSelector` in your Prometheus object
      role: 'alert-rules',
    },
    name: 'custom-rules',
    namespace: 'my-namespace',  // keep in mind that this namespace must be selected by your `ruleNamespaceSelector` in your Prometheus object
  },
  spec: {
    groups: [
      {
        name: 'example-group',
        rules: [
          {
            alert: 'Watchdog',
            expr: 'vector(1)',
            labels: {
              severity: 'none',
            },
            annotations: {
              description: 'This is a Watchdog meant to ensure that the entire alerting pipeline is functional.',
            },
          },
        ],
      },
    ],
  },
}

[borg286]

There is a valid use case for developers/ops to want special rules that generate composite metrics, but are not necessarily alerts. Right now we need to specify the labels of these rules as 'alert-rules' which may not be an appropiate label for general rules we want run. For example we may want to have SLO calculations that require lots of composition that consumes data from the last month. Having all these calculations performed at runtime would be cost-prohibitive.

I'd like to propose that the prometheus operator and custom resource definition allow for multiple rule selectors, so that it pulls in the PrometheusRules labeled with alert-rules, as well as some custom label like 'global-rules'. This would also enable us to have a prometheus that does aggregation at a global level, and a prometheus server acting on datacenter-level data, and a prometheus server acting on a large subset of services, and yet another acting on a subset of endpoints. Each prometheus server could be pulling in some general rule set, as well as an alert-rules specific to their granularity, as well as another rule set that aren't technically alerts but are rules that we want ran at that level.

Also I'd like some jsonnet library that makes writing a PrometheusRule easier. You've given an example of some json expanded out, which is ok. But I'd like to only need to write the labels and set of additional expressions.

[brancz]

I'd like to propose that the prometheus operator and custom resource definition allow for multiple rule selectors, so that it pulls in the PrometheusRules labeled with alert-rules, as well as some custom label like 'global-rules'. This would also enable us to have a prometheus that does aggregation at a global level, and a prometheus server acting on datacenter-level data, and a prometheus server acting on a large subset of services, and yet another acting on a subset of endpoints. Each prometheus server could be pulling in some general rule set, as well as an alert-rules specific to their granularity, as well as another rule set that aren't technically alerts but are rules that we want ran at that level.

You can already do this! :) The Prometheus resource has the ruleSelector and ruleNamespaceSelector, which you can use to build exactly what you described. The label I had in the example I provided, was just that, an example, you can use whatever label scheme you want! :)

Also I'd like some jsonnet library that makes writing a PrometheusRule easier. You've given an example of some json expanded out, which is ok. But I'd like to only need to write the labels and set of additional expressions.

Jsonnet is a turning complete language, you can totally build any abstraction that fits your organization's workflow!