Skip to content

Commit

Permalink
Actually drop container_network metrics (#453)
Browse files Browse the repository at this point in the history
We added container_network.* to the wrong part of the values so they are
still the highest-cardinality metrics (around 15% of all metrics).

Most of the new lines are copy-pasted from the default values.yaml in
the Chart, this change just adds container_network.

Tested by using `bazel build
src/app_charts/prometheus/prometheus-operator-chart.robot.yaml` to build
the YAML and diffing:

```
--- /tmp/before.yaml    2024-10-10 15:40:11.686509000 +0000
+++ /tmp/after.yaml     2024-10-10 15:52:02.595166797 +0000
@@ -1192,7 +1192,7 @@
       sourceLabels:
       - __name__
     - action: drop
-      regex: container_spec.*
+      regex: container_spec.*|container_network.*
       sourceLabels:
       - __name__
     - action: drop
```

then manually applying this change with k9s to a lab system and
verifying that the metric stops flowing to the cloud.
  • Loading branch information
drigz authored Oct 11, 2024
1 parent 9efafce commit 2d1b7e5
Showing 1 changed file with 20 additions and 0 deletions.
20 changes: 20 additions & 0 deletions src/app_charts/prometheus/prometheus-robot.values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -138,6 +138,26 @@ kubelet:
- action: drop
regex: "container_network.*|container_blkio.*|kubelet_.*|kubernetes_feature_enabled|container_fs.*|container_processes|container_last_seen|storage_operation_duration_seconds_bucket"
sourceLabels: [__name__]
cAdvisorMetricRelabelings:
# Drop high cardinality metrics from kubelet. (with metrics_path="/metrics/cadvisor")
- sourceLabels: [__name__]
action: drop
regex: 'container_cpu_(cfs_throttled_seconds_total|load_average_10s|system_seconds_total|user_seconds_total)'
- sourceLabels: [__name__]
action: drop
regex: 'container_fs_(io_current|io_time_seconds_total|io_time_weighted_seconds_total|reads_merged_total|sector_reads_total|sector_writes_total|writes_merged_total)'
- sourceLabels: [__name__]
action: drop
regex: 'container_memory_(mapped_file|swap)'
- sourceLabels: [__name__]
action: drop
regex: 'container_(file_descriptors|tasks_state|threads_max)'
- sourceLabels: [__name__]
action: drop
regex: 'container_spec.*|container_network.*'
- sourceLabels: [id, pod]
action: drop
regex: '.+;'
relabelings:
- sourceLabels: [__meta_kubernetes_pod_node_name]
targetLabel: instance
Expand Down

0 comments on commit 2d1b7e5

Please sign in to comment.