Unified gateway for multiple clusters

[turboFei]

Motivation

For some company, there might be multiple Hadoop cluster(data center)s.
Currently, a Kyuubi server only serves one cluster.
So multiple Kyuubi servers are needed for multiple clusters.
It seems meaningful to enable Kyuubi as a unified gateway for multiple clusters.

Restriction

To unify multiple clusters, there are some restrictions.

1. The node that kyuubi service is on should be able to access all these clusters.

2. The authentication for these clusters should be consistent. For example, if the authentication method is KERBEROS, for these clusters, they need have the same KDC realm or the KDC are trusted for each other. If the authentication method is LDAP, the ldap server endpoint for these clusters should be the same.

3. There should be a service discovery(for kyuubi server side) that is visible for all these clusters.
   For example, if the service discovery is by ZooKeeper, all these clusters should be able to access this Zookeeper service.
   If Kyuubi is deployed on k8s, they can use k8s load balancer(name service) as service discovery space for kyuubi server side.

Solution

For Kyuubi on k8s use case(use k8s load balancer for kyuubi server discovery):

1. Add a new config item to specify cluster name by client.
2. Config the environment variables for different clusters in kyuubi service side.
3. Support disable service discovery in kyuubi server side and expose engine instance to the cluster zookeeper storage.
4. For different cluster, choose relate hadoop/hive/spark(flink etc.)/zk quorum conf to connect/create engine instance.

For other use case, they need have a service discovery space that could be accessible by all the clusters.

MISC

How do you think about this?

Is it general or just a specific use case?

[yaooqinn]

Currently, a Kyuubi server only serves one cluster.

It can work with multiple Hadoop clusters with the ability that Spark provides in the same engine, IIUC.

Or do you mean specifying different HADOOP_CONF_DIR per engine? I remember that you have already provided an option called kyuubi.egnineEnv.XX to make it happen ?

[turboFei]

Currently, a Kyuubi server only serves one cluster.

It can work with multiple Hadoop clusters with the ability that Spark provides in the same engine, IIUC.

Or do you mean specifying different HADOOP_CONF_DIR per engine? I remember that you have already provided an option called kyuubi.egnineEnv.XX to make it happen ?

Thanks for your reminder, it seems can work.
But client side need transfer plenty of configuration items.

So it seems that we can treat the clusterName as a SELECTOR.

It seems that we can extend the kyuubi.egnineEnv.XX to kyuubi.engineEnv.${SELECTOR}.**.
We can expose these pre-defined engine env variables with different SELECTOR for customers.
So that client side just need to transfer the SELECTOR then Kyuubi Server can choose related env variables.

[turboFei]

Besides the kyuubi.engineEnv.${SELECTOR}.**., kyuubi server also need get related zk quorum to connect related engine discovery space and find out existing engine instances.

So, maybe a config item likes kyuubi.engineVar.${SELECTOR}.** is also needed.

[pan3793]

Sounds like an engine template functionality.

[turboFei]

yes, just likes the selector concept in k8s.

[pan3793]

3. ... and expose engine instance to the cluster zookeeper storage.

What's the cluster zookeeper storage means?

[turboFei]

1. ... and expose engine instance to the cluster zookeeper storage.

What's the cluster zookeeper storage means?

I mean that, for example:

• cluster a
• cluster b

The engine instance for cluster a can register a zk node in cluster a.

And the engine instance for cluster b can register a zk node in cluster b.

[yaooqinn]

Thanks for your reminder, it seems can work.
But client side need transfer plenty of configuration items.

Spark support read all configuration file, e.g. spark ones, hive-site.xml, core-site.xml, hdfs-site.xml, etc in SPARK_CONF_DIR, so we can map one SPARK_CONF_DIR to one hadoop cluster. HADOOP_CONF_DIR can be an empty one.

The engine instance for cluster a can register a zk node in cluster a.
And the engine instance for cluster b can register a zk node in cluster b.

I this case, we need to not only pass the engine namespace to the engine side but also a session-level zookeeper ensemble address and build a zk client for this zk ensemble. Maybe it can work already, I am not quite sure.

[pan3793]

If Kyuubi is deployed on k8s, they can use k8s load balancer(name service) as service discovery space for kyuubi server side.

If I recall correctly, LB should be K8s vendor dependent

[turboFei]

In ebay, I deploy kyuubi in two k8s clusters. For each k8s cluster, I create a load balancer.

Then create federationAccessPoint to access these two k8s clusters.

And at last, create a nameservice as the final Load balancer for Kyuubi.

I am not sure whether it is the same experience for community k8s.