feat(prometheus): Switch to upstream prometheus as the default Prometheus image

[jsirianni]

Description of Changes

Some users have strict security requirements that require auditing of unapproved images, such as ghcr.io/observiq/bindplane-prometheus. This is a source of friction when prom/prometheus is already approved for use.

This PR replaces bindplane-prometheus with prom/prometheus. The end result is identical, as the bindplane-prometheus image is just a wrapper around prom/prometheus, with our configurations baked in.

The following changes were required

• Addition of configmap for injecting the three configuration files via volume mounts
• Entrypoint command args

This change will be transparent to the user. The latest and past few BindPlane releases used Prometheus v2.54.1, which is now the default prometheus.image.tag value. The configuration and entrypoint command remain unchanged.

Testing

When the image is deployed, it has the following files in /etc/prometheus.

/prometheus $ ls -la /etc/prometheus
total 24
drwxr-xr-x    1 nobody   nobody        4096 Sep 19 16:35 .
drwxr-xr-x    1 root     root          4096 Sep 19 16:35 ..
lrwxrwxrwx    1 nobody   nobody          39 Aug 27 11:20 console_libraries -> /usr/share/prometheus/console_libraries
lrwxrwxrwx    1 nobody   nobody          31 Aug 27 11:20 consoles -> /usr/share/prometheus/consoles/
-rw-r--r--    1 root     nobody          59 Sep 19 16:35 prometheus.yml
-rw-r--r--    1 root     nobody         629 Sep 19 16:35 rules.yml
-rw-r--r--    1 root     nobody         172 Sep 19 16:35 web.yml

If you compare this to bindplane-prometheus, it is similar.

/prometheus $ ls -la /etc/prometheus
total 20
drwxr-xr-x    1 nobody   nobody        4096 Aug 28 13:21 .
drwxr-xr-x    1 root     root          4096 Sep 19 16:45 ..
lrwxrwxrwx    1 nobody   nobody          39 Aug  9 12:03 console_libraries -> /usr/share/prometheus/console_libraries
lrwxrwxrwx    1 nobody   nobody          31 Aug  9 12:03 consoles -> /usr/share/prometheus/consoles/
-rw-------    1 nobody   nobody          59 Aug 28 13:21 prometheus.yml
-rw-------    1 nobody   nobody         629 Aug 28 13:21 rules.yml
-rw-------    1 nobody   nobody         172 Aug 28 13:21 web.yml

The file permissions are different, but that is okay. In this case, 0644 owned by root is sufficient for the Prometheus process to read the config files. We had more control over this before when we built our own images. We have less flexibility here when mounting config files into containers using volume mounts.

Default / StatefulSet

I tested by deploying the "default" configuration.

# values.yaml
config:
  username: bpuser
  password: bppass
  sessions_secret: 4484766F-5016-4077-B8E0-0DE1D637854B

helm upgrade \
  --install bindplane \
  ./charts/bindplane \
  --values values.yaml

NAME                                         READY   STATUS    RESTARTS   AGE
bindplane-0                                  1/1     Running   0          17m
bindplane-prometheus-0                       1/1     Running   0          94s
bindplane-transform-agent-84d4bf89f7-b527n   1/1     Running   0          17m

After configuring a node agent, measurements begin flowing in.

High Availability w/ Upgrade and Downgrade

I tested HA with NATS + Postgres just to be certain.

k apply -f test/helper/postgres/postgres.yaml 
k create secret generic bindplane --from-literal=license=$BINDPLANE_LICENSE       

# Requires a /etc/hosts entry and `minikube tunnel`
minikube addons enable ingress 

config:
  username: bpuser
  password: bppass
  sessions_secret: 4484766F-5016-4077-B8E0-0DE1D637854B
  server_url: http://bindplane.local:80
  # The secret "bindplane" should exist and have the
  # key license with a license key as the value.
  # License is required in CI in order to use Postgres store.
  licenseUseSecret: true

ingress:
  enable: true
  host: bindplane.local
  class: nginx

backend:
  type: postgres
  postgres:
    host: postgres.postgres.svc.cluster.local
    database: bindplane
    username: postgres
    password: password
    maxConnections: 20
    
eventbus:
  type: nats

replicas: 1

resources:
  requests:
    memory: 100Mi
    cpu: 100m
  limits:
    memory: 100Mi

nats:
  resources:
    requests:
      memory: 100Mi
      cpu: 100m
    limits:
      memory: 100Mi

helm upgrade \
  --install bindplane-ha \
  ./charts/bindplane \
  --values values.yaml

NAME                                            READY   STATUS    RESTARTS   AGE
bindplane-ha-77b886bf96-nffnf                   1/1     Running   0          52s
bindplane-ha-jobs-6fdbf56784-snwxz              1/1     Running   0          7s
bindplane-ha-nats-0                             1/1     Running   0          2m17s
bindplane-ha-nats-1                             1/1     Running   0          2m17s
bindplane-ha-nats-2                             1/1     Running   0          2m17s
bindplane-ha-prometheus-0                       1/1     Running   0          2m17s
bindplane-ha-transform-agent-777cb688db-cfvdb   1/1     Running   0          2m17s

I can switch back and forth between Charts bindplane/bindplane (latest release) and ./charts/bindplane (this branch) without issue. Simulating upgrades and downgrades.

Topology, Agents page, and agent health all look good.

Please check that the PR fulfills these requirements

• Tests for the changes have been added (for bug fixes / features)
• Docs have been added / updated (for bug fixes / features)
• CI passes
• Changes to ports, services, or other networking have been tested with istio

[jsirianni]

We used this helper to set the Prometheus image based on the Chartfile's application version (BindPlane's version). This is no longer nessisary because the values file has a default value for prometheus.image.tag.

[jsirianni]

These configurations match our specification, which can be found here or within the BindPlane repository under package/prometheus.

[jsirianni]

These arguments match the args defined here.