Skip to content

Commit

Permalink
Ensure default Spark executor SA is created in Spark clusters (#3921)
Browse files Browse the repository at this point in the history
This is mostly a disaster-recovery thing since this SA exists already -
but to avoid any unpleasant times should we decide to start these
clusters from scratch, it'd be best to ensure that this SA actually
exists :)
  • Loading branch information
nemacysts authored Jul 17, 2024
1 parent ce58341 commit 5d000db
Showing 1 changed file with 11 additions and 4 deletions.
15 changes: 11 additions & 4 deletions paasta_tools/setup_tron_namespace.py
Original file line number Diff line number Diff line change
Expand Up @@ -85,15 +85,22 @@ def ensure_service_accounts(job_configs: List[TronJobConfig]) -> None:
kube_client=kube_client,
)
# spark executors are special in that we want the SA to exist in two namespaces:
# the tron namespace - for the spark driver
# and the spark namespace - for the spark executor
if action.get_executor() == "spark":
# the tron namespace - for the spark driver (which will be created by the ensure_service_account() above)
# and the spark namespace - for the spark executor (which we'll create below)
if (
action.get_executor() == "spark"
# this should always be truthy, but let's be safe since this comes from SystemPaastaConfig
and action.get_spark_executor_iam_role()
):
# this kubeclient creation is lru_cache'd so it should be fine to call this for every spark action
spark_kube_client = KubeClient(
config_file=system_paasta_config.get_spark_kubeconfig()
)
# this will look quite similar to the above, but we're ensuring that a potentially different SA exists:
# this one is for the actual spark executors to use. if an iam_role is set, we'll use that, otherwise
# there's an executor-specifc default role just like there is for the drivers :)
ensure_service_account(
action.get_iam_role(),
action.get_spark_executor_iam_role(),
namespace=spark_tools.SPARK_EXECUTOR_NAMESPACE,
kube_client=spark_kube_client,
)
Expand Down

0 comments on commit 5d000db

Please sign in to comment.