Multiple Clusters on a single machine

[prodrigues-prodigentia]

Question / Where do you need Help?

How can I run multiple clusters at the same time (that contain a web application) on a single machine?

I want to have multiple clusters running at the same time, so I can just go to the browser and simply write https://test.localhost or https://test2.localhost and go to different web applications (each cluster has one application.)

Currently I have two clusters and both of them work if I use the common ports (80 and 443) of the host machine in the cluster.yml file. The way to access the applications is working properly with the help of Acrylix (I'm using Windows/WSL) but I'm not able to run both clusters at the same time since it doesn't let me start a cluster if the port he needs to use is already allocated (which it is, since it's being used by the other cluster):

FATA[0019] Failed to add one or more helper nodes: runtime failed to start node 'k3d-test-serverlb': docker failed to start container for node 'k3d-test-serverlb': Error response from daemon: driver failed programming external connectivity on endpoint k3d-test-serverlb (5b5764a4f7304dadedfeea161b3cf45863e17c0e180d30e48dccf1dbf00191d6): Bind for 0.0.0.0:443 failed: port is already allocated

After that I tried, even if it's not ideal, to run the second cluster with different host ports but for some reason I'm not being able to use the application in the browser (even if I specify the port with for example https://test.localhost:81.)

The cluster.yml file is like the following (the other one is basically the same but with a different name, host and hostPort):

apiVersion: k3d.io/v1alpha4
kind: Simple
metadata:
  name: test
servers: 1
agents: 3
kubeAPI:
  host: "test.localhost"
  hostIP: "127.0.0.1"
  hostPort: "6446"
image: rancher/k3s:v1.23.5-k3s1
volumes:
  - volume: \\wsl$\Ubuntu\home\user\dev\test:/var/www/code/
    nodeFilters:
      - agent:*
ports:
  - port: 80:80
    nodeFilters:
      - loadbalancer
  - port: 443:443
    nodeFilters:
      - loadbalancer
  - port: 27017:27017
    nodeFilters:
      - loadbalancer
  - port: 8081:8081
    nodeFilters:
      - loadbalancer
registries:
  use:
    - k3d-registry.localhost:5000
options:
  k3d:
    wait: true
  kubeconfig:
    updateDefaultKubeconfig: true
    switchCurrentContext: true

After creating the cluster with the previously shown configuration I install the cert-manager:

• kubectl create namespace cert-manager
• helm repo add jetstack https://charts.jetstack.io
• helm repo update
• helm install cert-manager --namespace cert-manager --version v1.5.4 --set installCRDs=true --set nodeSelector."kubernetes.io/os"=linux jetstack/cert-manager

And then apply the following manifests:

### APPLICATION ###

apiVersion: apps/v1
kind: Deployment
metadata:
  name: php-nginx-deployment
spec:
  selector:
    matchLabels:
      app: php-nginx
  replicas: 3
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: php-nginx
    spec:
      restartPolicy: Always
      containers:
        - name: nginx-php8
          image: k3d-registry.localhost:5000/test:test
          imagePullPolicy: Always
          ports:
            - name: http
              containerPort: 8080
          volumeMounts:
            - name: volume
              mountPath: /var/www/code
      volumes:
        - name: volume
          hostPath:
            path: /var/www/code
            type: Directory

---

apiVersion: v1
kind: Service
metadata:
  name: php-nginx-service
  labels:
    app: php-nginx-service
spec:
  ports:
    - port: 80
      name: http
      protocol: "TCP"
      targetPort: 8080
  selector:
    app: php-nginx
  sessionAffinity: None
  type: ClusterIP

---

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: php-nginx-ingress
spec:
  rules:
    - host: test.localhost
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: php-nginx-service
                port:
                  number: 80

---

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: [email protected]
    privateKeySecretRef:
      name: letsencrypt
    solvers:
      - http01:
          ingress:
            class: nginx
            podTemplate:
              spec:
                nodeSelector:
                  "kubernetes.io/os": linux

---

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: tls-secret
spec:
  secretName: tls-secret
  dnsNames:
    - test.localhost
  issuerRef:
    name: letsencrypt

--- ### MONGODB ###

apiVersion: apps/v1
kind: Deployment
metadata:
  name: mongo-deployment
spec:
  selector:
    matchLabels:
      app: mongo
  replicas: 1
  template:
    metadata:
      labels:
        app: mongo
    spec:
      containers:
        - name: mongo
          image: mongo:4.4.6
          ports:
            - containerPort: 27017
              protocol: TCP
          env:
            - name: MONGO_INITDB_ROOT_USERNAME
              value: "root"
            - name: MONGO_INITDB_ROOT_PASSWORD
              value: "rootPassword"

---

apiVersion: v1
kind: Service
metadata:
  name: mongo-service
  labels:
    app: mongo-service
spec:
  selector:
    app: mongo
  ports:
    - name: mongo
      port: 27017
      protocol: "TCP"
      targetPort: 27017
  sessionAffinity: None
  type: ClusterIP

---

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: mongo-ingress
spec:
  rules:
    - host: test.mongo.localhost
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: mongo-service
                port:
                  number: 27017

--- ### MONGO EXPRESS ###

apiVersion: apps/v1
kind: Deployment
metadata:
  name: mongo-express-deployment
spec:
  selector:
    matchLabels:
      app: mongo-express
  replicas: 1
  template:
    metadata:
      labels:
        app: mongo-express
    spec:
      containers:
        - name: mongo-express
          image: mongo-express:0.54.0
          ports:
            - containerPort: 8081
              protocol: TCP
          env:
            - name: ME_CONFIG_MONGODB_SERVER
              value: "mongo-service"
            - name: ME_CONFIG_MONGODB_PORT
              value: "27017"
            - name: ME_CONFIG_BASICAUTH_USERNAME
              value: "testUsername"
            - name: ME_CONFIG_BASICAUTH_PASSWORD
              value: "testPassword"
            - name: ME_CONFIG_MONGODB_ADMINUSERNAME
              value: "root"
            - name: ME_CONFIG_MONGODB_ADMINPASSWORD
              value: "rootPassword"

---

apiVersion: v1
kind: Service
metadata:
  name: mongo-express-service
  labels:
    app: mongo-express-service
spec:
  selector:
    app: mongo-express
  ports:
    - name: mongo-express
      port: 8081
      protocol: "TCP"
      targetPort: 8081
  sessionAffinity: None
  type: ClusterIP

---

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: mongo-express-ingress
spec:
  rules:
    - host: test.mongo-express.localhost
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: mongo-express-service
                port:
                  number: 8081

As I said before this works when it's just a single cluster, but I have not been able to make a second cluster work. If I try to use the same host ports it fails, if I try to use different ports it seems to not be accessible through the browser.

I would appreciate any help you can give, thank you!

Scope of your Question

• k3d version v5.4.4
• k3s version v1.23.8-k3s1 (default)

[prodrigues-prodigentia]

Update:

After that I tried, even if it's not ideal, to run the second cluster with different host ports but for some reason I'm not being able to use the application in the browser (even if I specify the port with for example https://test.localhost:81.)
It doesn't work because of "https." It works if I access "http://test.localhost:81."

Any help about how to be able to do it by not writing the port in the URL, and to have access to it by HTTPS is appreciated.

I'm liking this project a lot so far, good work, thank you!