Skip to content

Multi-cluster GitOps repository with Flux2, SOPS & Renovate

License

Notifications You must be signed in to change notification settings

Pumba98/flux2-gitops

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

k3s flux renovate

k8s clusters backed by Flux v2

Kubernetes clusters using the GitOps tool Flux.
The Git repository is the driving the state of the Kubernetes clusters.
The awesome Flux SOPS integration is used to encrypt secrets with age.

📂 Repository structure

The Git repository contains the following directories:

📁
├─📁 apps            # apps available for intallation
├─📁 cluster-apps    # kustomization and overlays for app installations per cluster
│  ├─📁 staging
│  └─📁 production
├─📁 charts          # helm chart repos
├─📁 configs         # configs per cluster
└─📁 base
   ├─📁 flux-system  # flux & gitops operator
   ├─📁 staging      # flux configuration per cluster
   └─📁 production   # flux configuration per cluster

💻  Software

The following apps are installed on the clusters.

Software Purpose
Flux2 GitOps Tool managing the cluster
Longhorn Persistent Block Storage Provisioner
NGINX Ingress Controller Cluster Ingress controller
MetalLB Bare metal LoadBalancer
Cert-Manager Letsencrypt certificates with Cloudflare DNS
ExternalDNS Configure Cloudflare DNS Servers
kube-vip Virtual IP Load-Balancer for Control Plane High Availability
Kube-Prometheus Stack Prometheus & Exporters to monitor the cluster
Grafana Monitoring & Logging Dashboard
Alertmanager Monitoring Alerts
Grafana Loki Log aggregation system
System Upgrade Controller Automated k3s upgrades
Weave GitOps Powerful WebUI extension to Flux for deployment insights
MinIO Amazon S3 compatible high Performance Object Storage
Authelia SSO & 2FA authentication server for Cluster Web Apps
Paperless-ngx Document management system
Tandoor-Recipes Recipe Manager and Meal Planner
File Browser Web File Browser
SFTPGo Full-featured SFTP Server
Immich Photo and video management solution
Memos A privacy-first, lightweight note-taking service
Syncthing Continuous file synchronization service
Radicale CalDAV and CardDAV Server
Rancher Kubernetes Management Dashboard
Homer Static dashboard for the cluster applications
Bind Full-featured DNS System
Blocky Fast and lightweight DNS proxy as ad-blocker
Pod-Gateway Route traffic through a VPN gateway
Descheduler Evicts pods to optimize scheduling
Goldilocks Utility to help identifying good resource requests and limits
X.509 Certificate Exporter A Prometheus exporter to monitor x509 certificates
SMB CSI Driver CSI Driver for SMB
kured Kubernetes Reboot Daemon (only used for Monitoring)
Kubernetes Metrics Server Source of container resource metrics for Kubernetes

🤖  Automation

Renovate Bot makes sure the components are never outdated.

It creates PullRequests when Helm charts or Docker images have newer versions available and even keeps Flux and k3s up-to-date.

🤝  Thanks

Big shout out to k8s@home and everyone from awesome-home-kubernetes for the inspiration ❤️

📖  Notes

📍 Installation Notes

tl;dr

kubectl create namespace flux-system --dry-run=client -o yaml | kubectl apply -f -
sops -d ./base/flux-system/init/flux-sops-age-secret.sops.yaml | kubectl apply -f -
sops -d ./base/flux-system/init/flux-secret.sops.yaml | kubectl apply -f -
kubectl apply --kustomize=./base/flux-system
kubectl apply --kustomize=./base/staging
  1. Pre-create the flux-system namespace
kubectl create namespace flux-system --dry-run=client -o yaml | kubectl apply -f -
  1. Add the Flux age key in-order for Flux to decrypt SOPS secrets
sops -d ./base/flux-system/init/flux-sops-age-secret.sops.yaml | kubectl apply -f -
  1. (Optional) Add the Flux SSH key in-order for Flux to pull private git repositories
sops -d ./base/flux-system/init/flux-secret.sops.yaml | kubectl apply -f -
  1. Install Flux
kubectl apply --kustomize=./base/flux-system
  1. Configure Flux
kubectl apply --kustomize=./base/staging
🚧 PostgreSQL Major Version Upgrade

I now use a tianon/postgres-upgrade init-container for PostgreSQL Major Upgrades.

Always take backups, dataloss is possible. Old data gets removed and replaced by output of pg_upgrade.

## Init Container for Major PostgreSQL Upgrades, not needed permanently
initContainers:
- name: pg-upgrade
  image: tianon/postgres-upgrade:15-to-16
  securityContext:
    runAsUser: 0
    runAsGroup: 0
  volumeMounts:
  - name: data
    mountPath: /bitnami/postgresql
  env:
  - name: "PG_OLD"
    value: "15"
  - name: "PG_NEW"
    value: "16"
  command:
  - /bin/bash
  - -c
  - |
    if [[ $(< /bitnami/postgresql/data/PG_VERSION) -eq $PG_NEW ]]; then echo "PostgreSQL is already up2date"; exit 0; fi
    if [[ "$PGBINOLD" != "/usr/lib/postgresql/$PG_OLD/bin" ]]; then echo "Wrong postgres-upgrade image"; exit -1; fi
    if [[ "$PGBINNEW" != "/usr/lib/postgresql/$PG_NEW/bin" ]]; then echo "Wrong postgres-upgrade image"; exit -1; fi
    echo "Upgrading PostgreSQL from $PG_OLD to $PG_NEW"
    cp -r /bitnami/postgresql/data /var/lib/postgresql/$PG_OLD
    usermod -u 1001 postgres
    groupmod -g 1001 postgres
    chown -R postgres:postgres /var/lib/postgresql
    su postgres -c 'PGDATA="$PGDATANEW" eval "initdb $POSTGRES_INITDB_ARGS"'
    cp -p $PG_NEW/data/postgresql.conf $PG_NEW/data/pg_hba.conf $PG_OLD/data/
    chmod 700 $PG_OLD/data
    gosu postgres pg_upgrade
    gosu postgres pg_ctl -D /var/lib/postgresql/$PG_NEW/data -l logfile start
    gosu postgres /usr/lib/postgresql/$PG_NEW/bin/vacuumdb --all --analyze-in-stages
    gosu postgres pg_ctl -D /var/lib/postgresql/$PG_NEW/data -l logfile stop
    rm /var/lib/postgresql/$PG_NEW/data/pg_hba.conf /var/lib/postgresql/$PG_NEW/data/postgresql.conf
    rm -rf /bitnami/postgresql/data
    mv /var/lib/postgresql/$PG_NEW/data /bitnami/postgresql/
🚧 Previous Manual Upgrade Approach

Based on bitnami/charts#1798 (comment)

  1. Preparation
export NAMESPACE=selfhosted
export APPLICATION_DEPLOYMENT=paperless-ngx
export POSTGRES_DEPLOYMENT=${APPLICATION_DEPLOYMENT}-postgresql
export POSTGRES_MAJOR_VERSION=12
export POSTGRES_PVC_SIZE=4Gi
export POSTGRES_DB=${APPLICATION_DEPLOYMENT}
export POSTGRES_USERNAME=${APPLICATION_DEPLOYMENT}
export POSTGRES_PASSWORD=yourSecretPassword

Note: POSTGRES_MAJOR_VERSION is the helm version, not postgresql version.

  1. Scale down application that uses the database
kubectl scale deployment ${APPLICATION_DEPLOYMENT} -n ${NAMESPACE} --replicas 0
  1. Deploy new major version of the database
helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo update

helm install ${POSTGRES_DEPLOYMENT}-upgrade bitnami/postgresql --version ${POSTGRES_MAJOR_VERSION} -n ${NAMESPACE} --wait \
    --set auth.username=${POSTGRES_USERNAME} \
    --set auth.password=${POSTGRES_PASSWORD} \
    --set auth.database=${POSTGRES_DB} \
    --set primary.persistence.size=${POSTGRES_PVC_SIZE}
  1. Migrate data to new postgresql deployment
kubectl exec -it ${POSTGRES_DEPLOYMENT}-upgrade-0 -n ${NAMESPACE} -- bash -c "export PGPASSWORD=${POSTGRES_PASSWORD}; time pg_dump -h ${POSTGRES_DEPLOYMENT} -U ${POSTGRES_USERNAME} | psql -U ${POSTGRES_USERNAME}"

From here on you have multiple possibilities, e.g. just use your app with the new db deployment.

I personally prefer to backup the volume of the new DB, uninstall both database deployments & delete their PVCs.

After that I restore the backup of the PVC with the name of the old database deployment & upgrade my Helmrelease version.

helm uninstall ${POSTGRES_DEPLOYMENT}-upgrade -n ${NAMESPACE}
kubectl scale sts ${POSTGRES_DEPLOYMENT} -n ${NAMESPACE} --replicas 0

The volume deletion and restore is done in Longhorn UI. Afterwards helm upgrade for the postgresql deployments can be done.

  1. Scale up application that uses the database
kubectl scale deployment ${APPLICATION_DEPLOYMENT} -n ${NAMESPACE} --replicas 1

About

Multi-cluster GitOps repository with Flux2, SOPS & Renovate

Topics

Resources

License

Stars

Watchers

Forks