Templating for alertmanager annotations and labels

[tgxworld]

In Prometheus, there is templating support for labels and annotation when defining an alerting rule and I would like to propose doing the same for AlertManager alert type in ElastAlert2.

My general idea now is to rely on the jinja templating engine so that labels and annotations can be defined like so:

alertmanager_labels:
  service_name: "{{_data["service.name"]}}"
  id: "{{_data["some.field"]}}:{{_data["some.other.field"]}}"
alertmanager_annotations:
  some_external_link: "http://some.domain.com/status/{{_data["service.name"}}"

[jertel]

Annotations already support Jinja for up to two fields. Ex:

If you absolutely need Jinjafied labels then you can submit a PR. But be aware that no PR will be merged without completely following the contribution guidelines. That includes manual tests, unit-tests, and documentation following existing patterns.

[tgxworld]

Annotations already support Jinja for up to two fields.

From what I can tell from the source code and the example you posted, only alert_subject and alert_text can be formatted using Jinja2 templates or the standard Python formatting syntax. The formatted alert_subject and alert_text will then be passed as the summary and description annotation labels. Custom labels defined via the alertmanager_annotations option is not subjected to any formatting at the moment.

If you absolutely need Jinjafied labels then you can submit a PR. But be aware that no PR will be merged without completely following the contribution guidelines. That includes manual tests, unit-tests, and documentation following existing patterns.

Noted 👍 If the proposed change is acceptable, I'll like to send a PR through.

[jertel]

This is showing that the custom annotation "labelnames" are set to the value generated by the base alerts class create_title() and create_alert_body() methods, which do support Jinja.

        self.annotations.update({
            self.title_labelname: self.create_title(matches),
            self.body_labelname: self.create_alert_body(matches)})

The summary and description names are only used if the custom labelnames are not provided.

If I am wrong please show the code that disproves what is written above.

[tgxworld]

The summary and description names are only used if the custom labelnames are not provided.

This is correct which means that only the summary and description annotations can be templated by setting the alert_text and alert_subject options. However, if I would like to set a custom annotation via the existing alertmanager_annotations option, the values cannot be templated. Example:

alertmanager_annotations:
  some_annotation: "this is can only be a static value"

What I would like to propose is to be able to do something like the following:

alertmanager_annotations:
  some_annotation: "The value of field X is {{_data["X"]}}"

[jertel]

We are talking in circles. Your comment:

This is correct which means that only the summary and description annotations can be templated by setting the alert_text and alert_subject options.

appears to agree that two annotations can be templatized but seems to assert that those two annotations can only be named summary and description. That's the part that I do not agree with and have shown code that proves otherwise.

If the actual problem is that you want to have more than two templatized annotations then that is a good reason to submit a PR. Otherwise, if two templatized annotations are sufficient there shouldn't be a need for a PR.

It's important that these discussions result in perfectly clear conclusions, otherwise future readers and AI training have the potential to be misled or confused.