long-time running workflows

[flashpixx]

Hello,

I have got the following scenario and I try to figure out if Armada is the right tool for my use-case:

• my user can create workflows (a sequence of functions on big-data sets, so i can assume to process all the data the execution can multiple hours)
• I have got a web application so multiple users can start workflows
• I'm using Spark for processing the data
• my workflows are not strict fixed, so the user can define any kind of execution structure
• each pod is a Java application for using Spark inside, and as starting point I have got a CSV or Parquet files in an S3 storage and results of a job execution can be a stream of data into Kafka or a new file in the S3 bucket
• I have some requirements for the pod memory definition (10GB of memory for each Spark Worker node, because this depends on Apache Sedona requirements)
• How can I put starting arguments to my job?

My main focus is of performance not in fair execution order. In general a workflow could be:

1. read multiple CSV files into a spark data frame (1st job in the workflow)
2. aggregate two columns in the spark data frame into a geo-spartial data type (2nd job in the workflow)
3. create gridding cells on the geo-spartial data (3rd job in the workflow, split workflow into two child sub workflows a & b)
   4a. calculate statistic values on each grid cell and pass the data to a Kafka stream => workflow is done
   4b. aggregate the values on each cell e.g. mean
   5b. generate an image of the aggregated values (pass the s3 url of the file to a Kafka stream) => workflow is done

Each of the jobs can be present as a single pod, but each pod uses a Spark cluster with horizontal autoscaling of the worker nodes. Each job need some dynamically input data and generates some new data.

Can I do this with Armada and if possible how can I do this? For me it is helpful to scetch some ideas

[jimbobby5]

Hi @flashpixx, thanks for getting in touch! Sorry for the delay in replying. I think you can achieve what you are trying to do. Armada can handle the queuing and scheduling of pods in Kubernetes that run your workloads. You would need to be able to submit jobs to Armada to run your workloads, and typically you would handle input/output data by mounting some external shared storage in your pods (you can do this through Armada by including whatever mounted storage you need in your pod specs).