feat(prometheus): Support StatefulSet with Prometheus sidecar

[jsirianni]

Description of Changes

The Helm chart needs to be enhanced to support the latest measurement backend (Prometheus). We can deploy Prometheus as a sidecar when operating BindPlane as a single pod StatefulSet.

• Persistent volume claim for /var/lib/prometheus/tsdb
• Configmap with volume mounts to /etc/prometheus

Future PRs will include the following

1. Basic Auth
2. TLS
3. Advanced options such as queryPrefix and remote write (for using Mimir or other advanced architectures)

Testing

Tested by deploying to minikube.

minikube start
helm template charts/bindplane --values test/cases/prometheus_sidecar/values.yaml | kubectl apply -f -

Setup port forwarding.

kubectl port-forward pod/release-name-bindplane-0 3011:3001

kubectl port-forward pod/release-name-bindplane-0 -c prometheus 9999:9090

Connect to bindplane at http://localhost:3011

Connect to prometheus at http://localhost:9999

Create a k8s node config and deploy the agent. The agent will connect to bindplane without issue because Helm's generate remote URL is correct for the minikube cluster.

Topology view will show measurements working.

The Prometheus UI will have metrics. The rollup metric will be found at bindplane_agent_measurements:rollup:rate:1m after waiting a few minutes.

You can exec into prometheus with

kubectl exec -it release-name-bindplane-0 -c prometheus sh

The configs will be in /etc/prometheus and the storage will be at /var/lib/prometheus/tsdb.

ps aux will show the following command

/bin/prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --web.config.file=/etc/prometheus/web.yml \
  --storage.tsdb.retention.time=2d \
  --web.enable-remote-write-receiver \
  --web.listen-address=127.0.0.1:9090 \
  --storage.tsdb.path=/var/lib/prometheus/tsdb \
  --web.console.templates=/etc/prometheus/consoles \
  --web.console.libraries=/etc/prometheus/console_libraries

Please check that the PR fulfills these requirements

• Tests for the changes have been added (for bug fixes / features)
• Docs have been added / updated (for bug fixes / features)
• CI passes