Skip to content

Latest commit

 

History

History
81 lines (64 loc) · 5.46 KB

alerts.md

File metadata and controls

81 lines (64 loc) · 5.46 KB

Alerts

This page contains the steps to create alerts for the Observability for Kubernetes Operator.

Table of Content

Alert Templates

We have alert templates on common Kubernetes issues.

Alert Template
Detect if observability status is unhealthy observability-status-unhealthy.json.tmpl
Detect pod stuck in pending pod-stuck-in-pending.json.tmpl
Detect pod stuck in terminating pod-stuck-in-terminating.json.tmpl
Detect pod backoff event pod-backoff-event.json.tmpl
Detect workload with non-ready pods workload-not-ready.json.tmpl
Detect pod out-of-memory kills pod-out-of-memory-kills.json.tmpl
Detect container cpu throttling container-cpu-throttling.json.tmpl
Detect container cpu overutilization container-cpu-overutilization.json.tmpl
Detect persistent volumes with no claims persistent-volumes-no-claim.json.tmpl
Detect persistent volumes with error persistent-volumes-error.json.tmpl
Detect persistent volumes filling up persistent-volume-claim-overutilization.json.tmpl
Detect node memory overutilization node-memory-overutilization.json.tmpl
Detect node cpu overutilization node-cpu-overutilization.json.tmpl
Detect node filesystem overutilization node-filesystem-overutilization.json.tmpl
Detect node cpu-request saturation node-cpu-request-saturation.json.tmpl
Detect node memory-request saturation node-memory-request-saturation.json.tmpl
Detect node disk pressure condition node-disk-pressure.json.tmpl
Detect node memory pressure condition node-memory-pressure.json.tmpl
Detect node condition not ready node-not-ready.json.tmpl
Detect etcd has no leader etcd-no-leader.json.tmpl

Creating Alerts

  1. Ensure that you have the information for the required fields:
    • Wavefront API token. See Managing API Tokens page.
    • Wavefront instance. For example, the value of <YOUR_WAVEFRONT_INSTANCE> from your wavefront url (https://<YOUR_WAVEFRONT_INSTANCE>.wavefront.com).
    • Cluster name. For example, the value of clusterName from your Wavefront Custom Resource configuration (ex: mycluster-us-west-1).
    • (Optional) Alert template. For example, the value of <alert_template_file.json.tmpl> from the list of alert templates (ex: pod-backoff-event.json.tmpl).
    • (Optional) Alert target. For example, an email address, PagerDuty key, or alert target. Alert targets can be a comma separated list.

Example: Creating All the Alerts

curl -sSL https://raw.githubusercontent.com/wavefrontHQ/observability-for-kubernetes/main/docs/alerts/create-all-alerts.sh | bash -s -- \
  -t <YOUR_API_TOKEN> \
  -c <YOUR_WAVEFRONT_INSTANCE> \
  -e <YOUR_ALERT_TARGET> \
  -n <YOUR_CLUSTER_NAME>

Note: You will need to change <YOUR_API_TOKEN>, <YOUR_WAVEFRONT_INSTANCE>, <YOUR_ALERT_TARGET>, and <YOUR_CLUSTER_NAME> in the above example.

Example: Creating a Single Alert

curl -sSL https://raw.githubusercontent.com/wavefrontHQ/observability-for-kubernetes/main/docs/alerts/create-alert.sh | bash -s -- \
  -t <YOUR_API_TOKEN> \
  -c <YOUR_WAVEFRONT_INSTANCE> \
  -n <YOUR_CLUSTER_NAME> \
  -e <YOUR_ALERT_TARGET> \
  -f <ALERT_TEMPLATE>

Note: You will need to change <YOUR_API_TOKEN>, <YOUR_WAVEFRONT_INSTANCE>, <YOUR_CLUSTER_NAME>, <YOUR_ALERT_TARGET>, and <ALERT_TEMPLATE> in the above example.

Customizing Alerts

  1. Log in to your service instance https://<YOUR_WAVEFRONT_INSTANCE>.wavefront.com as a user with the Alerts permission. Click Alerting > All Alerts from the toolbar to display the Alerts Browser.
  2. Click the alert name, or click the ellipsis icon next to the alert and select Edit. You can search for the alert by typing the alert name in the search field.
  3. Change the alert properties when you edit the alert.
  4. Click Save in the top right to save your changes.

Note: See Create and Manage Alerts for an overview on how to create and manage alerts.