HA for thrift interface (binary/http) via nginx

[anandnalya]

Hi,

I've deployed three instance of Kyuubi behind nginx. With this setup, I'm able to submit jobs through REST (BatchSession) but when connecting to thrift interface (both binary, and HTTP), the client fails to connect.

Does thrift interface supports HA through load balancer? I can't use Zookeeper based service discovery and the clients don't have permission to connect to Zookeeper.

[pan3793]

The THRIFT BINARY protocol is based on TCP protocol, should be fine to use w/ nginx. But THRIFT HTTP should not work, since different Kyuubi instances do not share session state

[anandnalya]

Thanks for the response @pan3793. I tried THRIFT binary through nginx, and when only one upstream Kyuubi instance is defined it works.

When I define three upstream servers with the following conf:

  upstream _ky_thrift_binary {
    hash $remote_addr;
    server ky.meta1.hiyu.hadoop.dfw.wordpress.com:10009;
    server ky.meta2.hiyu.hadoop.dfw.wordpress.com:10009;
    server ky.meta3.hiyu.hadoop.dfw.wordpress.com:10009;
  }

  server {
    listen 10009;
    proxy_pass _ky_thrift_binary;
  }

I run into the following exception:

2023-06-05 04:55:27.717 INFO ZookeeperDiscoveryClient: Get service instance:data-C12-30.hiyu.hadoop.dfw.wordpress.com:1489 engine id:application_1685695355449_0025 and version:1.7.1 under /kyuubi_1.7.1_USER_SPARK_SQL/anandnalya/default
2023-06-05 04:55:27.726 WARN KyuubiSessionImpl: Failed to open [kyuubi_USER_SPARK_SQL_anandnalya_default_3ec5b106-0683-441d-85db-daf3d4bb4140 data-C12-30.hiyu.hadoop.dfw.wordpress.com:1489] after 0/9 times, retrying
java.net.ConnectException: Connection refused (Connection refused)
	at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:?]
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:412) ~[?:?]
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:255) ~[?:?]
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:237) ~[?:?]
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:?]
	at java.net.Socket.connect(Socket.java:609) ~[?:?]
	at org.apache.thrift.transport.TSocket.open(TSocket.java:221) ~[libthrift-0.9.3.jar:0.9.3]
	at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:266) ~[libthrift-0.9.3.jar:0.9.3]
	at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37) ~[libthrift-0.9.3.jar:0.9.3]
	at org.apache.kyuubi.client.KyuubiSyncThriftClient$.createTProtocol(KyuubiSyncThriftClient.scala:456) ~[kyuubi-server_2.12-1.7.1.jar:1.7.1]
	at org.apache.kyuubi.client.KyuubiSyncThriftClient$.createClient(KyuubiSyncThriftClient.scala:472) ~[kyuubi-server_2.12-1.7.1.jar:1.7.1]
	at org.apache.kyuubi.session.KyuubiSessionImpl.$anonfun$openEngineSession$2(KyuubiSessionImpl.scala:155) ~[kyuubi-server_2.12-1.7.1.jar:1.7.1]
	at org.apache.kyuubi.session.KyuubiSessionImpl.$anonfun$openEngineSession$2$adapted(KyuubiSessionImpl.scala:123) ~[kyuubi-server_2.12-1.7.1.jar:1.7.1]
	at org.apache.kyuubi.ha.client.DiscoveryClientProvider$.withDiscoveryClient(DiscoveryClientProvider.scala:36) ~[kyuubi-ha_2.12-1.7.1.jar:1.7.1]
	at org.apache.kyuubi.session.KyuubiSessionImpl.$anonfun$openEngineSession$1(KyuubiSessionImpl.scala:123) ~[kyuubi-server_2.12-1.7.1.jar:1.7.1]
	at org.apache.kyuubi.session.KyuubiSession.handleSessionException(KyuubiSession.scala:49) ~[kyuubi-server_2.12-1.7.1.jar:1.7.1]
	at org.apache.kyuubi.session.KyuubiSessionImpl.openEngineSession(KyuubiSessionImpl.scala:123) ~[kyuubi-server_2.12-1.7.1.jar:1.7.1]
	at org.apache.kyuubi.operation.LaunchEngine.$anonfun$runInternal$2(LaunchEngine.scala:60) ~[kyuubi-server_2.12-1.7.1.jar:1.7.1]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[?:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
	at java.lang.Thread.run(Thread.java:829) ~[?:?]

I'll try to debug this on my end. FWIW, I'm running Kyuubi 1.7.1.

[pan3793]

I suppose you are using Spark cluster mode, and the data-C12-30.hiyu.hadoop.dfw.wordpress.com is a YARN Node Manager node. Is each Kyuubi instance's network reachable to this node?

Basically, when a new connection comes in, Kyuubi lookups for a suitable engine from the ZK, and if no candidates, it invokes spark-submit to launch a new one. The launched engine register itself to the ZK so that the Kyuubi can find them.

You should make sure that your network topology meets the above requirements.