Bitnami HelmChart resources failing

[kallaics]

Describe the bug

More of our HelmChart (generated automatically by FluxCD) are failing. Only the Bitnami based repositories affected. I found they are moving to OCI format, but looks like they are breaking the normal rules under FluxCD. As I see well, the Bitnami are not changing the repository format to OCI, just replace the chart source to OCI protocol. So HelmRepository with OCI chart source.

Same happened few days ago with RabbitMQ and now the MongoDB deployments.

In this case I got error message from Flux for many Bitnami based Helm charts:

flux-system	helmchart/flux-system-mongodb	16.3.0	False	False	chart pull error: failed to download chart for remote reference: Get "oci://registry-1.docker.io/bitnamicharts/mongodb:16.3.1": unsupported protocol scheme "oci"

I have no idea, it is a not implemented feature in the FluxCD or the Bitnami tries to do something non usual with their changes. I hope you will have idea, how it is possible to solve it. I am happy about any solution.

Thanks guys!

Steps to reproduce

1. Install FluxCD
2. Create a Bitnami HelmRepository (Stanadard and OCI type, because OCI type not works!)

apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: HelmRepository
metadata:
  name: bitnami
  namespace: flux-system
spec:
  interval: 30m
  type: default
  url: https://charts.bitnami.com/bitnami

4. Create a HelmRelease with sourceRef definition, that pointing to for example the Bitnami MongoDB Helm chart.

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: mongodb
  namespace: flux-system
spec:
  releaseName: mongodb
  targetNamespace: default
  chart:
    spec:
      chart: mongodb
      sourceRef:
        kind: HelmRepository
        name: bitnami
        namespace: flux-system
  interval: 1m
  install:
    remediation:
      retries: 3
  rollback:
    disableWait: true
    disableWaitForJobs: true
    disableHooks: true
    recreate: true
    force: true
    cleanupOnFail: true
  upgrade:
    remediation:
      retries: 3
      remediateLastFailure: true
      strategy: rollback
  values:
    architecture: standalone
    replicaCount: 1
    useStatefulSet: true

6. Helm chart will be failed, when the Chart version will be higher, than 16.3.0, the change comes with 16.3.1

Expected behavior

The created HelmChart should be support OCI connection (or the Bitnami implementation is not compatible with FluxCD)

Screenshots and recordings

No response

OS / Distro

MacOS, K8s clusters are using Ubuntu

Flux version

flux: v2.4.0

Flux check

► checking prerequisites
✔ Kubernetes 1.29.9 >=1.28.0-0
► checking version in cluster
✔ distribution: flux-v2.4.0
✔ bootstrapped: true
► checking controllers
✔ helm-controller: deployment ready
► ghcr.io/fluxcd/helm-controller:v1.1.0
✔ image-automation-controller: deployment ready
► ghcr.io/fluxcd/image-automation-controller:v0.39.0
✔ image-reflector-controller: deployment ready
► ghcr.io/fluxcd/image-reflector-controller:v0.33.0
✔ kustomize-controller: deployment ready
► ghcr.io/fluxcd/kustomize-controller:v1.4.0
✔ notification-controller: deployment ready
► ghcr.io/fluxcd/notification-controller:v1.4.0
✔ source-controller: deployment ready
► ghcr.io/fluxcd/source-controller:v1.4.1
► checking crds
✔ alerts.notification.toolkit.fluxcd.io/v1beta3
✔ buckets.source.toolkit.fluxcd.io/v1
✔ gitrepositories.source.toolkit.fluxcd.io/v1
✔ helmcharts.source.toolkit.fluxcd.io/v1
✔ helmreleases.helm.toolkit.fluxcd.io/v2
✔ helmrepositories.source.toolkit.fluxcd.io/v1
✔ imagepolicies.image.toolkit.fluxcd.io/v1beta2
✔ imagerepositories.image.toolkit.fluxcd.io/v1beta2
✔ imageupdateautomations.image.toolkit.fluxcd.io/v1beta2
✔ kustomizations.kustomize.toolkit.fluxcd.io/v1
✔ ocirepositories.source.toolkit.fluxcd.io/v1beta2
✔ providers.notification.toolkit.fluxcd.io/v1beta3
✔ receivers.notification.toolkit.fluxcd.io/v1
✔ all checks passed

Git provider

Gitlab Enterprise (self-hosted)

Container Registry provider

DockerHub

Additional context

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

[kallaics]

I found the an issue about the changes by Bitnami.

bitnami/charts#30110

[kallaics]

I found the solution:

Add OCIRepository for each Helm chart

apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: OCIRepository
metadata:
  name: bitnami-mongodb
spec:
  interval: 30m
  url: oci://registry-1.docker.io/bitnamicharts/mongodb
  ref:
    semver: ">=16.3.1" # First available version of the software in OCIRepository

Next step to update the HelmRelease source:
Remove spec/chart section and add the spec/chartRef section on the following way.

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: mongodb
  namespace: flux-system
spec:
  releaseName: mongodb
  targetNamespace: default
  chartRef:
    kind: OCIRepository
    name: bitnami-mongodb
    namespace: flux-system
  interval: 1m
  install:
    remediation:
      retries: 3
  rollback:
    disableWait: true
    disableWaitForJobs: true
    disableHooks: true
    recreate: true
    force: true
    cleanupOnFail: true
  upgrade:
    remediation:
      retries: 3
      remediateLastFailure: true
      strategy: rollback
  values:
    architecture: standalone
    replicaCount: 1
    useStatefulSet: true

[fgeck]

I am trying to run bitnami postgres and following the recommended steps.

Do you run into this as well?

bitnami     oci://registry-1.docker.io/bitnamicharts/postgres   False     failed to determine the artifact tag for 'oci://registry-1.docker.io/bitnamicharts/postgres': GET https://registry-1.docker.io/v2/bitnamicharts/postgres/tags/list?n=1000: UNAUTHORIZED: authentication required; [map[Action:pull Class: Name:bitnamicharts/postgres Type:repository]]   16m

Do I need docker pull secrets? I thought this is a public registry.

Repo:

---
# yaml-language-server: $schema=https://raw.githubusercontent.com/fluxcd-community/flux2-schemas/refs/heads/main/ocirepository-source-v1beta2.json
apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: OCIRepository
metadata:
  name: bitnami
  namespace: flux-system
spec:
  interval: 12h
  url: oci://registry-1.docker.io/bitnamicharts/postgres
  ref:
    semver: "^16.4.3"

[kallaics]

Looks like you made a simple typo. The URL, based on the documentation is not correct (need postgresql instead of postgres at the end of the URL)

Documentation: https://artifacthub.io/packages/helm/bitnami/postgresql/16.4.3

Your OCIRepository resource should be like this after URL fix:

---
# yaml-language-server: $schema=https://raw.githubusercontent.com/fluxcd-community/flux2-schemas/refs/heads/main/ocirepository-source-v1beta2.json
apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: OCIRepository
metadata:
  name: bitnami
  namespace: flux-system
spec:
  interval: 12h
  url: oci://registry-1.docker.io/bitnamicharts/postgresql
  ref:
    semver: "^16.4.3"

[fgeck]

Thank you very much! Classic mistake...

[stefanprodan]

I suggest you name the OCIRepository as postgresql-chart, you'll probably deploy more things from the Bitnami catalog and repos will override each other if the name collides.

[kallaics]

@stefanprodan I used this (old) way too and I just refreshed URL and I added the type: oci to HelmRepository resource.
In this case he can also keep the way to deploy one HelmRepository and can deploy from multiple Helm charts from this source. (The source in this case will be HelmRepository instead of OCIRepository by HelmRelease.)
Will this way support for long term?

apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
  name: bitnami
  namespace: flux-system
spec:
  interval: 12h
  url: oci://registry-1.docker.io/bitnamicharts
  type: oci

Update: Based on @stefanprodan 's reply, this way is not recommend to use ( for Bitnami repositories).

[stefanprodan]

Using a HelmRepo means you can't reuse a chart across multiple releases, even if you deploy the same postgresql 10 times, you'll pull it ten times from bitnami and get rate limited, instead of just one time (what you can do with OCIRepo).

[kallaics]

Thank you @stefanprodan !
I agree 100% with you!