Thanos rule does not add labels to recording rules

[atos-cosmin-gavagiuc]

What happened:

Labels added to recording rules through PrometheusRule are not visible in Thanos rule, even though they are present in the container configuration and appear in the Thanos API.

What you expected to happen:

Add labels to recording rules added to Thanos Rule as specified in the Recording Rules documentation

How to reproduce it (as minimally and precisely as possible):
Recording rule:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  labels:
    monitoring: thanos-monitored
  name: up-rules
spec:
  groups:
  - name: thanos-rule-test
    rules:
    - expr: |
        up
      record: alpha:up
      labels:
        release_stage: alpha

The recording rule is correctly loaded in Thanos, including the lables specified in the configuration.

The labels declared in the recording rule are missing the Thanos rules dashboard and from the metric labels.

Full logs to relevant components:

Anything else we need to know:
Thanos version 0.35.1

[dosubot]

Hey @atos-cosmin-gavagiuc, I'm here to help you with any bugs, questions, or contributions you have. Let's tackle this issue together!

Thanos Rule processes and applies labels from PrometheusRule configurations by allowing you to specify labels that can be added or overwritten before storing the result of rule evaluations. This is done within the rule configuration files using the labels field. However, if the labels are not appearing in the Thanos rules dashboard or the metric labels, it might be due to how Thanos Rule handles label inheritance or overwriting.

Ensure that the external labels specified when running Thanos Rule do not overwrite the labels defined in your recording rules. External labels are crucial for distinguishing high-availability pairs and ensuring that the data is correctly identified in a multi-cluster setup. If these external labels are set, they might replace the source labels, affecting the visibility of your custom labels.

Additionally, verify that the Thanos Querier is correctly configured to fan-out to all known Rules endpoints and that the deduplication process is not affecting the visibility of your labels. The deduplication logic involves several levels, including rule group level, rule type, rule name, rule label names, and rule expression fields, which might impact how labels are displayed ^[1][2].

To continue talking to Dosu, mention @dosu.

[yeya24]

Hi @atos-cosmin-gavagiuc, I am trying to reproduce the issue myself.

I can confirm that labels are missing in Thanos Ruler UI so this is something we need to fix. Recording rule metrics have the correct label present so this is not an issue. Can you please double check? How did you know that your metric doesn't have the correct label. If the label is really missing probably some set up issue on your querier.

[atos-cosmin-gavagiuc]

Hi @yeya24 , I can confirm that the recording rule now has the correct label. I'm not sure why it was missing initially, perhaps I didn't wait long enough.