Distributed multi-cluster management

[mxq-151]

in my company,we have multi bigdata platform,it is not friendly for data-developer to use scheduling tool,they need to login to different cluster. to make the work easy ,the arch in the picture desc as below:
1.only have one auth server and auth-db
2.data-developer choose the ds-cluster-xx to use in ds-ui
3.afer choose ds-cluster-xx,ds-ui will send request to ds-api in ds-cluster-xx
4.schdule job/task/alert will run only in there own cluster

in one word,Cluster isolation is to ensure that scheduling does not affect each other, and it can also ensure scalability

[mxq-151]

any suggestion about this idea?
@SbloodyS

[SbloodyS]

According to your scene, I would suggest to deploy according to the above architecture. Deploy different worker-servers on different clusters. And using the project management of DS to manage different cluster differentiated projects.

In this way, new clusters can be deployed in a lightweight manner. @mxq-151

[mxq-151]

i consider the architecture your describe ,there's a problem :can db or master hold ten thousands task ?and the network maybe Unreliable,does it take affected to running task?
@SbloodyS

[SbloodyS]

I've successfully run more than 100000 tasks in the production environment every day with 32c 128G mysql and 16c 64g high-available master-server. So there is no problem with your concern.

However, not only DS clusters, such as Hadoop clusters, but also unreliable network is a fatal problem in all big data clusters. May I ask why the network unreliable is in the production environment?

[mxq-151]

I've successfully run more than 100000 tasks in the production environment every day with 32c 128G mysql and 16c 64g high-available master-server. So there is no problem with your concern.

However, not only DS clusters, such as Hadoop clusters, but also unreliable network is a fatal problem in all big data clusters. May I ask why the network unreliable is in the production environment?

it's reliable in cluster but the network between the clusters can not be promised

[SbloodyS]

it's reliable in cluster but the network between the clusters can not be promised

An alternative is IEPL/IPLC. The stability of the network is necessary.

[mxq-151]

thank you for your share