Investigate utilizing discovery.kubelet to handle super-large clusters

[petewall]

On very large clusters, using pod discovery:

discovery.kubernetes "pods" {
  role = "pods"
}

Can sometimes flood the amount of memory required for discovery and the amount of requests to the API server.

An interesting approach would be to set alloy-metrics to be a daemonset and utilize:

discovery.kubelet "pods" {
  ...
}

This limits Alloy's pod discovery to its local node only.

Doc: https://grafana.com/docs/alloy/latest/reference/components/discovery/discovery.kubelet/