Can we connect to external Spark Service

[singh-b]

My organisation already has a big data platform with Spark services running. Theoretically I could connect to existing Spark services for ingestion job, is that correct understanding?

[woop]

Hey @singh-b!

What system are you using to manage Spark jobs?

[singh-b]

Thanks for quick turnaround @woop
We use YARN for hadoop infra. I see many large scale companies ( atleast few known to me) have invested heavily in YARN and it would be good to support that, in case it does not work in the current state.

[woop]

We don't currently support YARN. We currently support the following Spark job managers https://github.com/feast-dev/feast-spark/tree/master/python/feast_spark/pyspark/launchers

We'd love to have support for YARN, but it's unlikely that we will implement it soon since we already support SparkOp for open source deployments. We'd be happy to receive a PR if you'd like to take a stab at it!