Impersonation of Kyuubi Spark Authz plugin

[AnirudhVyas]

How does spark ranger plugin migrated to Kyuubi work for impersonation?

will such a code work when using Kyuubi:

val ugi = UserGroupInformation.loginUsingKeytab(blah blah)
val proxy = UserGroupInformation.createProxyUser("some.user", ugi)
UsergroupInformation.setLoginUser(ugiProxy)

given that keytab file was generated for super user - would this impersonation work so that kyuubi sends audit logs to elastic search (ranger audits) as the proxy user?

[pan3793]

it works on client mode, but not cluster mode

[yaooqinn]

/**
   * Get the active session user
   * @param spark spark context instance
   * @return the user name
   */
  def getAuthzUgi(spark: SparkContext): UserGroupInformation = {
    // kyuubi.session.user is only used by kyuubi
    val user = spark.getLocalProperty("kyuubi.session.user")
    if (user != null && user != UserGroupInformation.getCurrentUser.getShortUserName) {
      UserGroupInformation.createRemoteUser(user)
    } else {
      UserGroupInformation.getCurrentUser
    }
  }

the snippet above is how the plugin handles users. IIUC, you can set the proxy user via kyuubi.session.user as a spark context local property

[AnirudhVyas]

I wish to get this going on EMR on EKS - if we ignore EMR on EKS and just consider spark on kubernetes - do you @yaooqinn or @pan3793 think it can work?

[pan3793]

The issue only occurs under the following conditions:

1. Kyuubi and Spark are kerberized
2. and Ranger is kerberized
3. and Kyuubi uses --proxy-user mode (default behavior) instead of --keytab to submit Spark Application
4. and submit Spark Application in cluster mode.

[pan3793]

It's because that Ranger does not support using the delegate token for authentication, under the above conditions, the Ranger client is running inside the Driver process w/o keytab file for authentication, the solution is to ship a dedicated ranger keytab on submitting phase, we have done this for zookeeper, you can learn more details by searching keywords kyuubi.ha.zookeeper.auth.keytab in the codebase.

[fuhaiq]

It's because that Ranger does not support using the delegate token for authentication, under the above conditions, the Ranger client is running inside the Driver process w/o keytab file for authentication, the solution is to ship a dedicated ranger keytab on submitting phase, we have done this for zookeeper, you can learn more details by searching keywords kyuubi.ha.zookeeper.auth.keytab in the codebase.

Hi, we are facing the exactly same problem in cluster mode by using the delegate token for authentication in ranger. Could you please explain in detail how to use 'kyuubi.ha.zookeeper.auth.keytab' ? Is there any document ? thanks!

[pan3793]

@fuhaiq, to be clear, kyuubi.ha.zookeeper.auth.keytab can not solve your Ranger issue, it's a solution for Zookeeper, but the idea could be reused for Ranger.

how to use 'kyuubi.ha.zookeeper.auth.keytab'

1. Generate a dedicated keytab for zookeeper authentication, it means, even the keytab leaks, will not cause a major security issue.
2. Add kyuubi.ha.zookeeper.auth.keytab=</path/of/the/zookeeper/dedicated/keytab> in kyuubi-defaults.conf

Then, Server & Engine will use this keytab to access Zookeeper instead of the superuser keytab (Server) and delegation token (Engine).

[fuhaiq]

@fuhaiq, to be clear, kyuubi.ha.zookeeper.auth.keytab can not solve your Ranger issue, it's a solution for Zookeeper, but the idea could be reused for Ranger.

how to use 'kyuubi.ha.zookeeper.auth.keytab'

1. Generate a dedicated keytab for zookeeper authentication, it means, even the keytab leaks, will not cause a major security issue.
2. Add kyuubi.ha.zookeeper.auth.keytab=</path/of/the/zookeeper/dedicated/keytab> in kyuubi-defaults.conf

Then, Server & Engine will use this keytab to access Zookeeper instead of the superuser keytab (Server) and delegation token (Engine).

Thanks! Get it

[AnirudhVyas]

We made it to work on EMR on EKS by using Kyuubi authz plugin as spark sql extension and adding all jars on spark usr lib and setting appropriate hadoop configurations. Essentially we did this:
0. Customize EMR EKS build image

1. we needed core-site xml to specify ldap groups mapping goes to /usr/lib/spark/conf or /etc/spark/conf or /etc/hadoop/conf
2. We added ranger-* conf goes to emr eks conf (same as above)
3. we then updated to provide capability to run kerberos kinit on container on build image
4. finally when we run we use above boiler plate code to tell spark who logs in
   and then do

val ugiProxy = ugi.createProxyUser("username", loggedInUserUgi)
ugiProxy.doAs( spark sql goes here)

All stuff works with above changes, Posting for other folks if they hit this issue or need.

[amanraj2520]

Where was this boiler plate code done @AnirudhVyas in the Spark code or Kyuubi code?

[xiaozhch5]

Hello, I used Apache Ranger to control the access policy of the Hadoop cluster, and kyuubi sync the policy of hive. When I config the properties using the instroduction of "https://kyuubi.apache.org/docs/latest/security/authorization/spark/install.html", some problem arised:

The log shows that, It's not certified by Ranger Admin.

So I changed the code of org.apache.ranger.admin.client.RangerAdminRESTClient, add username and password to be authenticated by Ranger Admin.

Both on client mode and cluster mode, kyuubi spark can successful sync the policy from Ranger Admin.

[amanraj2520]

@AnirudhVyas @yaooqinn @pan3793 @xiaozhch5 I am currently trying to submit a spark sql job through zeppelin (using livy interpreter) to Spark Engine. I am just using the Kyuubi Authz module as an extension while initializing the spark session. Note that I am not using any other module of Kyuubi. My Ranger and Spark engine is Kerberos enabled.

Kyuubi version - 1.8.0
Spark version - 3.3.1

While submitting a query from Zeppelin to Spark, livy impersonates the domain user and submits the spark-sql job as --proxy-user

Is there no way to pass the delegation token of the domain user when RangerAdminRESTClient tries to authenticate to Ranger. I am also getting the same error as mentioned by @xiaozhch5 and I am blocked on this for a couple of weeks. Pasted below.

Can someone please help here. I am very new to Kyuubi and Spark. Will really appreciate it.

[pan3793]

as mentioned here, there are some conditions to trigger this issue, from your description, I think you match the 1~3, do you exactly match the 4?

[amanraj2520]

Hi @pan3793 Thanks a lot for your quick reply. Yes I match 4. The query is submitted in cluster mode.

[amanraj2520]

Also the ranger policies are getting applied correctly if I launch a spark-sql session on the cluster which essentially means client mode.

[pan3793]

@amanraj2520 please read the convenience thoroughly, it requires basic knowledge of the Kerberos mechanism and Hadoop security system to understand this issue, I had tried my best to explained the detail of this issue.

TBH, Kerberos mechanism and Hadoop security system is complex topic, explaining it from the scratch needs to write a book.

[pan3793]

to be clear, all words "engine" or "kyuubi engine" could be replaced with "spark driver" in the context.