GKE Fast api, Redis, Celery, Flower, Rabbit

[Bellota22]

Hi everyone,

Lately, I've been trying to deploy my services in GKE. The context of the problem is that I have a long process and want to add it to a task queue. When I run it in Docker, everything works just fine. Then, I tried to do it in Minikube, built some services and deployments, and it worked fine as well. However, when I tried to deploy it in GKE, all services are up and running, but my Flower doesn't seem to recognize my Celery worker. I've been doing some research and ended up looking into Gateway and Ingress.

I'm new to Kubernetes and GKE, so it would be helpful to have some insights from you guys. Thanks!

These are my deployments and services. I'm not using Ingress just yet.

# celery-worker-deployment.yml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: celery-worker
spec:
  replicas: 1
  selector:
    matchLabels:
      app: celery-worker
  template:
    metadata:
      labels:
        app: celery-worker
    spec:
      containers:
        - name: celery-worker
          image: bellota23/test1:my-api
          args: 
            - 'poetry'
            - 'run'
            - 'celery'
            - '-A'
            - 'app.worker.celery_worker'
            - 'worker'
            - '-l'
            - 'info'
            - '-Q'
            - 'test-queue'
            - '-c'
            - '1'
          env:
            - name: RABBITMQ_HOST
              value: rabbitmq
            - name: REDIS_HOST
              value: redis

#celery-worker-hpa.yml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: celery-worker-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: celery-worker
  minReplicas: 1
  maxReplicas: 3
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 80

#fastapi-deployment.yml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: fastapi
  
spec:
  replicas: 1
  selector:
    matchLabels:
      app: fastapi
  template:
    metadata:
      labels:
        app: fastapi
    spec:
      containers:
      - name: fastapi
        image: bellota23/test1:my-api
        ports:
        - containerPort: 8000
        args: ["poetry", "run", "hypercorn", "app/main:app", "--bind", "0.0.0.0:8000", "--reload"]
        env:
        - name: RABBITMQ_HOST
          value: rabbitmq
        - name: REDIS_HOST
          value: redis

#fastapi-service.yml
apiVersion: v1
kind: Service
metadata:
  name: fastapi-service
  
spec:
  selector:
    app: fastapi
  type: LoadBalancer
  ports:
    - protocol: TCP
      port: 8000
      targetPort: 8000

#flower-deployment.yml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: celery-flower
  
spec:
  replicas: 1
  selector:
    matchLabels:
      app: celery-flower
  template:
    metadata:
      labels:
        app: celery-flower
    spec:
      containers:
        - name: celery-flower
          image: gregsi/latest-celery-flower-docker:latest
          ports:
            - containerPort: 5555
          env:
            - name: AMQP_ADMIN_HOST
              value: rabbitmq
            - name: AMQP_ADMIN_USERNAME
              value: user
            - name: AMQP_ADMIN_PASSWORD
              value: bitnami
            - name: AMQP_ADMIN_PORT
              value: "15672"
            - name: AMQP_HOST
              value: rabbitmq
            - name: AMQP_PASSWORD
              value: bitnami
            - name: AMQP_PORT
              value: "5672"
            - name: AMQP_USERNAME
              value: user
            - name: FLOWER_BASIC_AUTH
              value: user:test

#flower-service.yml
apiVersion: v1
kind: Service
metadata:
  name: celery-flower-service
  
spec:
  type: LoadBalancer
  selector:
    app: celery-flower
  ports:
    - protocol: TCP
      port: 5555
      targetPort: 5555