Overhead management for Kepler

[eklee15]

Questions/motivations

• When collecting Kepler measurements from multiple nodes/pods, how much is the overhead?
  • How many nodes/pods can we scale out/up? (Nodes/pods)
  • What would be the requirements for scalability for Kepler?
• Targeted components
  • Prometheus Server
  • Kepler exporter
  • Grafana Dashboards

Related issues/discussions

• Prometheus client scaling #439
• Introducing sampling method for Kepler and finding reasonable sampling rate #836
• ebpf: reducing ebpf call overhead by using sampling instead of tracing every calls #823

[mcalman]

Thanks for opening this discussion @eklee15. I have begun some overhead/scalability testing using a local Kind cluster. So far I have been using cAdvisor metrics exposed by Prometheus to measure overhead. Here are some initial results:

Scaling the size of a cluster with Kepler deployed
I created a Kind cluster with Docker container "nodes". I then used taints to deploy Kepler to a new node every 20 minutes and examined the overhead as the number of nodes with Kepler deployed increased. The plots below show the additional overhead consumed by the Prometheus container as Kepler is deployed on additional nodes.

Increasing the number of containers for Kepler to monitor
For this experiment, every 5 minutes I added an additional dummy container (Debian containers executing integer addition every 10ms) to a single node with Kepler deployed.

[mcalman]

@rootfs Here is the comparison @marceloamaral is referring to. The plots show Prometheus overhead that can be attributed to Kepler (I have subtracted out the average overhead values when Kepler is not deployed).

[rootfs]

this is cool! @mcalman can you summarize your recommendations and add to the project board

[mcalman]

@rootfs I believe I need to be granted access in order to add to the project board

[rootfs]

got it. Would you please create github issues and tasks? I'll link the project card to your issues/tasks

[mcalman]

Thanks @rootfs. Done #963 (comment)

[husky-parul]

@AdrianHammond @sthaha @vimalk78

[eklee15]

@rootfs I think this experiment will be more interesting when we see the resource saturation. We are trying to see 3 different aspects.

1. Number of dummy pod scaling (how many pods Kepler can cover if we increase the number of pods in BM/VM)
2. Number of nodes (vm) scaling (Kepler exporter can scale up to more than 100 or more nodes?)
3. ebpf sampling rate. (what is the reasonable sampling rate?)

[rootfs]

let's take this research into action
#964