cardinality type assistance

[jernster]

Hello, I'm trying to use elastalert to alert when our cluster hdfs capacity is less than a specific threshold (min_cardinality). I tried the cardinality type but it doesn't seem to be working. It's triggering alerts/matches even though it shouldn't as the cluster has much more capacity than the specified threshold. Am I using the appropriate type for this kind of alert? The more I try to work on this I'm not sure elastalert is necessarily the proper tool for this particular alert/use case but it's entirely possible I'm not using it correctly. Any tips/direction are appreciated and i'm happy to provide any additional information should it be useful - thank you!

My config is as follows:

name: HDFS Capacity Warning
type: cardinality
index: dev-index-*
timeframe:
    minutes: 5
filter:
  - query:
      query_string:
        query: "(service: hdfs) AND (metric_tag: dfs.FSNamesystem)"
query_key: CapacityRemaining
field_value: CapacityRemaining
cardinality_field: CapacityRemaining
# 15% of 23.07TB - development configured HDFS capacity
min_cardinality: 3460500000000
ignore_null: false
aggregation:
  minutes: 5
#realert:
#  minutes: 5
include:
  - CapacityRemaining
alert:
  - slack
alert_text_type: aggregation_summary_only
slack_username_override: "ElastAlert"
slack_emoji_override: "Warning Sign"
slack_msg_color: warning
slack_title: |-
        Converged Cluster

        Issue: HDFS Capacity Warning
        Environment: Dev

summary_table_fields:
  - CapacityRemaining

[jertel]

Cardinality is the count of unique values for a given field. That's probably not what you want. Check out the Metric Aggregation rule instead, using these inputs:

min_threshold: 3460500000000
metric_agg_type: avg
metric_agg_key: CapacityRemaining

Which would alert when you have less than 3.46TB remaining.

[jernster]

Oh wow, that's great. I'll give this a shot - I really appreciate your help and quick response. Thank you!

[jernster]

Hi @jertel, I marked your answer is correct as I think this is working now. I'm curious though if I'm missing something that shows the CapacityRemaining value in the actual alert?

Currently it shows up as None but would be nice to see the value there.

Aggregation resulted in the following data for summary_table_fields ==> ['CapacityRemaining', 'count']:

+-------------------+-------+
| CapacityRemaining | count |
+===================+=======+
| None              | 1     |
+-------------------+-------+

The config I ended up with:

name: HDFS Capacity Warning
type: metric_aggregation
index: dev-index-*
timeframe:
    minutes: 5
filter:
  - query:
      query_string:
        query: "(service: hdfs) AND (metric_tag: dfs.FSNamesystem)"
#query_key: CapacityRemaining
field_value: CapacityRemaining
metric_agg_key: CapacityRemaining
metric_agg_type: avg
# 15% of 23.07TB - development configured HDFS capacity
min_threshold: 8000000000000
#min_threshold: 3460500000000
ignore_null: false
aggregation:
  minutes: 5
include:
  - CapacityRemaining
alert:
  - slack
alert_text_type: aggregation_summary_only
slack_username_override: "ElastAlert"
slack_emoji_override: "Warning Sign"
slack_msg_color: warning
slack_title: |-
        Cluster

        Issue: HDFS Capacity Warning
        Environment: Dev

summary_table_fields:
  - CapacityRemaining

min_threshold currently set to trigger an alert.

Thanks again!

[jertel]

try:

summary_table_fields:
- metric_CapacityRemaining_avg

[jernster]

That was it. Thanks again for your assistance, I greatly appreciate it and now have a better understanding of how this is supposed to work.