pooling engine instances for better user query experience

[zinking]

the amount of time it takes to bootstrap a spark engine instance is currently innegligible. 30s at minimum.
it is possible to pool a few spark engine instances at the server and then later assigning to user when requests come in.

this will cause a certain a mount of resource waste, but with DRA and controlling the pool size , waste could be reduced.

[yaooqinn]

you can create a dummy connection for that user before he/she do the real work

[zinking]

don't really know who he/she is, I mean this could be a built-in mechanism for bringing engine up. would this be helpful ?

[yaooqinn]

the engine is user-oriented. mostly you need to start an engine for a specific user.

if you are not working with a secured cluster and want all engines to be shared across users, you can use GROUP or SERVER with engine-pool together.

[zinking]

sounds good. but is group/server level engine instance pooled already ? I thought they had only 1 instance as well and not pooled.

[yaooqinn]

yes, the engine instance is pooled(mostly), and kyuubi.engine.pool.size can help create multi instances inside that pool

[zinking]

thanks for the explaination

[pan3793]

Kyuubi provides a conf kyuubi.session.engine.launch.async, default is false, does it help?

[zinking]

I guess this should be the default implementation then.

and from the user's perspective, it is not async because now the engine start up times gets diluted in the calculation step.

guess I mis-understood @yaooqinn 's explanation as well. because I'm targeting shortening spark launch time here.

[pan3793]

You are right. async has been introduced in v1.4.0 and default enabled since v1.5.0

[pan3793]

I'm targeting shortening spark launch time here.

Are you using Spark on Yarn?

[zinking]

yes, I was suggesting pooling say 1 spark instance when kyuubi server starts, and when user comes in we allocate that instance to the user and refill that pool.

but this seems rather complicated in the current ha setup.

[pan3793]

You can pre-install Spark binaries on Yarn workers, and configure spark.yarn.jars, it can save time of uploading and distributing Spark jars of launching engine
https://spark.apache.org/docs/latest/running-on-yarn.html