baseline_of_kubernetes_container_network_io.yml

name: Baseline Of Kubernetes Container Network IO
id: 6edaca1d-d436-42d0-8df0-6895d3bf5b70
version: 4
date: '2024-09-24'
author: Matthew Moore, Splunk
type: Baseline
status: production
description: This baseline rule calculates the average and standard deviation of inbound
  and outbound network IO for each Kubernetes container. It uses metrics from the
  Kubernetes API and the Splunk Infrastructure Monitoring Add-on. The rule generates
  a lookup table with the average and standard deviation of the network IO for each
  container. This baseline can be used to detect anomalies in network communication
  behavior, which may indicate security threats such as data exfiltration, command
  and control communication, or compromised container behavior.
search: "| mstats avg(k8s.pod.network.io) as io where `kubernetes_metrics` by k8s.cluster.name
  k8s.pod.name k8s.node.name direction span=10s | eval service = replace('k8s.pod.name',
  \"-\\w{5}$|-[abcdef0-9]{8,10}-\\w{5}$\", \"\") | eval key = 'k8s.cluster.name' +
  \":\" + 'service' | stats avg(eval(if(direction=\"transmit\", io,null()))) as avg_outbound_network_io
  avg(eval(if(direction=\"receive\", io,null()))) as avg_inbound_network_io stdev(eval(if(direction=\"\
  transmit\", io,null()))) as stdev_outbound_network_io stdev(eval(if(direction=\"\
  receive\", io,null()))) as stdev_inbound_network_io count latest(_time) as last_seen
  by key | outputlookup k8s_container_network_io_baseline"
how_to_implement: "To implement this detection, follow these steps: 1. Deploy the
  OpenTelemetry Collector (OTEL) to your Kubernetes cluster. 2. Enable the hostmetrics/process
  receiver in the OTEL configuration. 3. Ensure that the process metrics, specifically
  Process.cpu.utilization and process.memory.utilization, are enabled. 4. Install
  the Splunk Infrastructure Monitoring (SIM) add-on (ref: https://splunkbase.splunk.com/app/5247)
  5. Configure the SIM add-on with your Observability Cloud Organization ID and Access
  Token. 6. Set up the SIM modular input to ingest Process Metrics. Name this input
  \"sim_process_metrics_to_metrics_index\". 7. In the SIM configuration, set the Organization
  ID to your Observability Cloud Organization ID. 8. Set the Signal Flow Program to
  the following: data('process.threads').publish(label='A'); data('process.cpu.utilization').publish(label='B');
  data('process.cpu.time').publish(label='C'); data('process.disk.io').publish(label='D');
  data('process.memory.usage').publish(label='E'); data('process.memory.virtual').publish(label='F');
  data('process.memory.utilization').publish(label='G'); data('process.cpu.utilization').publish(label='H');
  data('process.disk.operations').publish(label='I'); data('process.handles').publish(label='J');
  data('process.threads').publish(label='K') 9. Set the Metric Resolution to 10000.
  10. Leave all other settings at their default values."
known_false_positives: none
references: []
tags:
  analytic_story:
  - Abnormal Kubernetes Behavior using Splunk Infrastructure Monitoring
  detections:
  - Kubernetes Anomalous Inbound Outbound Network IO
  product:
  - Splunk Enterprise
  - Splunk Enterprise Security
  - Splunk Cloud
  security_domain: network
deployment:
  scheduling:
    cron_schedule: 0 2 * * 0
    earliest_time: -30d@d
    latest_time: -1d@d
    schedule_window: auto