Ozone doesn't work with Kerberos and Spark #4996

MerelyOneMax1986 · 2023-06-28T08:58:59Z

MerelyOneMax1986
Jun 28, 2023

We planned to go to production with Ozone , but the setup (Ozone 1.3.0 + Kerberos + Spark cluster + Kubernetes) does not work.

We can't go to the enterprise without Kerberos, regardless of having Ozone worked without Kerberos.

Should we go back to HDFS or is there some kind of solution?

MerelyOneMax1986 · 2023-06-28T09:52:05Z

MerelyOneMax1986
Jun 28, 2023
Author

The setup that doesn't work is the following: Apache Ozone 1.3.0 + Spark 3.2.3 (cluster mode) + Kerberos. All resides in Kubernetes 1.21.6.

Here you are Spark driver logs:
spark-driver.log

2 replies

MerelyOneMax1986 Jun 28, 2023
Author

The setup that works well is the following: HDFS 2.7.2 (HA mode) + Spark 3.2.3 (cluster mode) + Kerberos. All resides in Kubernetes 1.21.6.

Here you are Spark driver logs:
spark_driver_to_hdfs.log

MerelyOneMax1986 Jun 29, 2023
Author

Dear Ozone team, I am open to any kind of collaboration as Ozone is a cutting edge system that is becoming as a modern replacement for HDFS.

Currently, could you please share core-site.xml and hdfs-site.xml (and other meaningful conf) for Apache Ozone + Kerberos setup?

cloudcheflabs · 2023-07-01T07:30:58Z

cloudcheflabs
Jul 1, 2023

If kerberos authentication in spark is not mandatory, spark can access ozone using s3g which is s3 compatible.

For me, after making ozone secure with kerberos, I created access and secret key from s3g.
With these s3 credentials, spark can write to ozone, for instance, write iceberg table through s3g.

Kidong.

5 replies

MerelyOneMax1986 Jul 2, 2023
Author

@cloudcheflabs Thanks for you reply! Unfortunalely, it's not an option for our concept where we're currently using o3fs protocol as a replacement for hdfs:// and accordingly to the Ozone docs it should work once Ozone cluster has been secured.

Spark workers are able to write to s3g secured? Did you use Ozone HA in your installation? And anyway, could you please share your configs with us?

Looking forward to your reply! And we're hoping to collaborate.

cloudcheflabs Jul 2, 2023

Hello @MerelyOneMax1986 ,

I have installed ozone with kerberos security.
I have 3 SCM and 3 OM with HA and 3 Data Nodes.
In addition, I have 3 S3g.

3 S3g are mapped as reverse proxy entries to NGINX. For this, NGINX configuration looks like this.

upstream ozone-s3g {
    least_conn;
    server chango-private-1.chango.private:9878;
    server chango-private-2.chango.private:9878;
    server chango-private-3.chango.private:9878;
}

server {
    listen 80;
    listen  [::]:80;
    server_name  ozone-s3g.example.com;
    ignore_invalid_headers off;
    client_max_body_size 0;
    proxy_buffering off;
    proxy_request_buffering off;

    location / {
        proxy_set_header Host $http_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        proxy_connect_timeout 300;
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        chunked_transfer_encoding off;

        proxy_pass http://ozone-s3g;
   }
}

For me, Spark driver and executors of my spark job can read and write data to ozone through S3g. That is, Spark has nothing to do with kerberos authentication. Spark just knows S3 connection to S3G to access ozone with S3 credential which can be obtained with this command, for instance.

ozone s3 getsecret \
--om-service-id=chango \
-u ozone/[email protected] \
;

...
awsAccessKey=ozone/[email protected]
awsSecret=92584c79f4f360351bb392df37359610819c3a3d50695e6092910170c5db541f

Ok, I am going to share my ozone-site.xml and core-site.xml as below.

ozone-site.xml

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<configuration>
  <!-- om -->
  <property>
    <name>ozone.om.ratis.enable</name>
    <value>true</value>
  </property>
  <property>
    <name>ozone.om.service.ids</name>
    <value>chango</value>
  </property>
  <property>
    <name>ozone.om.nodes.chango</name>
    <value>om1,om2,om3</value>
  </property>
  <property>
    <name>ozone.om.address.chango.om1</name>
    <value>chango-private-1.chango.private</value>
  </property>
  <property>
    <name>ozone.om.address.chango.om2</name>
    <value>chango-private-2.chango.private</value>
  </property>
  <property>
    <name>ozone.om.address.chango.om3</name>
    <value>chango-private-3.chango.private</value>
  </property>
  <property>
    <name>ozone.om.address</name>
    <value>0.0.0.0:9862</value>
  </property>
  <property>
    <name>ozone.metadata.dirs</name>
    <value>/export/ozone/meta</value>
  </property>
  <property>
    <name>ozone.om.db.dirs</name>
    <value>/export/ozone/om-db</value>
  </property>
  <property>
    <name>ozone.om.rpc-port</name>
    <value>9862</value>
  </property>
  <property>
    <name>ozone.om.ratis-port</name>
    <value>9872</value>
  </property>
  <property>
    <name>ozone.om.http-port</name>
    <value>9874</value>
  </property>
  <property>
    <name>ozone.om.https-port</name>
    <value>9875</value>
  </property>
  <!-- scm -->
  <property>
    <name>ozone.scm.ratis.enable</name>
    <value>true</value>
  </property>
  <property>
    <name>ozone.scm.service.ids</name>
    <value>chango</value>
  </property>
  <property>
    <name>ozone.scm.nodes.chango</name>
    <value>scm1,scm2,scm3</value>
  </property>
  <property>
    <name>ozone.scm.address.chango.scm1</name>
    <value>chango-private-1.chango.private</value>
  </property>
  <property>
    <name>ozone.scm.address.chango.scm2</name>
    <value>chango-private-2.chango.private</value>
  </property>
  <property>
    <name>ozone.scm.address.chango.scm3</name>
    <value>chango-private-3.chango.private</value>
  </property>
  <property>
    <name>ozone.scm.client.address</name>
    <value>chango-private-1.chango.private</value>
  </property>
  <property>
    <name>ozone.scm.names</name>
    <value>chango-private-1.chango.private,chango-private-2.chango.private,chango-private-3.chango.private</value>
  </property>
  <property>
    <name>ozone.scm.datanode.id.dir</name>
    <value>/export/ozone/meta-node</value>
  </property>
  <property>
    <name>ozone.scm.db.dirs</name>
    <value>/export/ozone/scm-db</value>
  </property>
  <property>
    <name>ozone.scm.datanode.port</name>
    <value>9861</value>
  </property>
  <property>
    <name>ozone.scm.block.client.port</name>
    <value>9863</value>
  </property>
  <property>
    <name>ozone.scm.client.port</name>
    <value>9860</value>
  </property>
  <property>
    <name>ozone.scm.http-port</name>
    <value>9876</value>
  </property>
  <property>
    <name>ozone.scm.https-port</name>
    <value>9877</value>
  </property>
  <property>
    <name>ozone.scm.ratis.port</name>
    <value>9894</value>
  </property>
  <property>
    <name>ozone.scm.grpc.port</name>
    <value>9895</value>
  </property>
  <property>
    <name>ozone.scm.security.service.port</name>
    <value>9961</value>
  </property>
  <!-- dn -->
  <property>
    <name>hdds.datanode.dir</name>
    <value>/export/ozone/dn-data</value>
  </property>
  <property>
    <name>hdds.datanode.http-address</name>
    <value>0.0.0.0:9882</value>
  </property>
  <property>
    <name>dfs.datanode.data.dir</name>
    <value>/export/ozone/dn-data</value>
  </property>
  <property>
    <name>dfs.container.ratis.datanode.storage.dir</name>
    <value>/export/ozone/dn-db</value>
  </property>
  <property>
    <name>ozone.datanode.http-port</name>
    <value>9882</value>
  </property>
  <property>
    <name>ozone.datanode.https-port</name>
    <value>9883</value>
  </property>
  <property>
    <name>dfs.container.ratis.ipc</name>
    <value>9858</value>
  </property>
  <property>
    <name>dfs.container.ipc</name>
    <value>9859</value>
  </property>
  <property>
    <name>hdds.datanode.client.port</name>
    <value>9864</value>
  </property>
  <property>
    <name>dfs.container.ratis.datastream.port</name>
    <value>9855</value>
  </property>
  <property>
    <name>dfs.container.ratis.admin.port</name>
    <value>9857</value>
  </property>
  <property>
    <name>dfs.container.ratis.server.port</name>
    <value>9856</value>
  </property>
  <!-- s3g -->
  <property>
    <name>ozone.s3g.http-port</name>
    <value>9878</value>
  </property>
  <property>
    <name>ozone.s3g.https-port</name>
    <value>9879</value>
  </property>
  <!-- recon -->
  <property>
    <name>ozone.recon.rpc-port</name>
    <value>9891</value>
  </property>
  <property>
    <name>ozone.recon.http-port</name>
    <value>9888</value>
  </property>
  <property>
    <name>ozone.recon.https-port</name>
    <value>9889</value>
  </property>
  <property>
    <name>ozone.recon.sql.db.jooq.dialect</name>
    <value>MYSQL</value>
  </property>
  <property>
    <name>ozone.recon.sql.db.jdbc.url</name>
    <value>jdbc:mysql://chango-private-1.chango.private:3306/ozone_recon?useSSL=false&amp;createDatabaseIfNotExist=true</value>
  </property>
  <property>
    <name>ozone.recon.sql.db.username</name>
    <value>recon</value>
  </property>
  <property>
    <name>ozone.recon.sql.db.password</name>
    <value>Reconpass1#</value>
  </property>
  <property>
    <name>ozone.recon.sql.db.driver</name>
    <value>com.mysql.jdbc.Driver</value>
  </property>
  <property>
    <name>ozone.security.enabled</name>
    <value>true</value>
  </property>
  <property>
    <name>hadoop.security.authentication</name>
    <value>kerberos</value>
  </property>
  <property>
    <name>hdds.scm.kerberos.principal</name>
    <value>scm/[email protected]</value>
  </property>
  <property>
    <name>hdds.scm.http.auth.kerberos.principal</name>
    <value>scm_http/[email protected]</value>
  </property>
  <property>
    <name>hdds.scm.kerberos.keytab.file</name>
    <value>/etc/ozone/ozone.keytab</value>
  </property>
  <property>
    <name>ozone.om.kerberos.principal</name>
    <value>om/[email protected]</value>
  </property>
  <property>
    <name>ozone.om.http.auth.kerberos.principal</name>
    <value>om_http/[email protected]</value>
  </property>
  <property>
    <name>ozone.om.kerberos.keytab.file</name>
    <value>/etc/ozone/ozone.keytab</value>
  </property>
  <property>
    <name>ozone.s3g.kerberos.principal</name>
    <value>s3g/[email protected]</value>
  </property>
  <property>
    <name>ozone.s3g.http.auth.kerberos.principal</name>
    <value>s3g_http/[email protected]</value>
  </property>
  <property>
    <name>ozone.s3g.kerberos.keytab.file</name>
    <value>/etc/ozone/ozone.keytab</value>
  </property>
  <property>
    <name>dfs.datanode.kerberos.principal</name>
    <value>datanode/[email protected]</value>
  </property>
  <property>
    <name>hdds.datanode.http.auth.kerberos.principal</name>
    <value>datanode_http/[email protected]</value>
  </property>
  <property>
    <name>dfs.datanode.kerberos.keytab.file</name>
    <value>/etc/ozone/ozone.keytab</value>
  </property>
  <property>
    <name>hdds.datanode.http.auth.kerberos.keytab</name>
    <value>/etc/ozone/ozone.keytab</value>
  </property>
</configuration>

core-site.xml

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->
<!-- Put site-specific property overrides in this file. -->
<configuration>
  <property>
    <name>fs.ofs.impl</name>
    <value>org.apache.hadoop.fs.ozone.RootedOzoneFileSystem</value>
  </property>
  <property>
    <name>fs.defaultFS</name>
    <value>ofs://chango</value>
  </property>
  <property>
    <name>fs.trash.interval</name>
    <value>10</value>
  </property>
  <property>
    <name>fs.trash.classname</name>
    <value>org.apache.hadoop.ozone.om.TrashPolicyOzone</value>
  </property>
  <property>
    <name>hadoop.security.authentication</name>
    <value>KERBEROS</value>
  </property>
</configuration>

MerelyOneMax1986 Jul 3, 2023
Author

@cloudcheflabs, thanks a lot for sharing! Your answer gave a hope that OM works with Kerberos and Spark cluster.
I am going to thoroughly check the properties you pointed out, and I will get back to you once checked.

Let's be in touch! Have a nice day! Cheers)

cloudcheflabs Jul 4, 2023

@MerelyOneMax1986 , glad with your new success.

If you have installed SCM with HA with Kerberos without success, you may see this.

Problem with enabling ozone security #4956

I had also problem with ozone kerberos to start scm in correct order.

MerelyOneMax1986 Jul 4, 2023
Author

@cloudcheflabs thanks a lot! I really appreciate your help!
Hope that now I am on the right way, on the way to success.

Let's stay in touch!

jojochuang · 2023-08-07T18:32:33Z

jojochuang
Aug 7, 2023
Collaborator

The most relevant log is these two lines

23/06/12 04:35:13 DEBUG OzoneDelegationTokenSelector: Got tokens: null for service 10.246.0.181:9862
23/06/12 04:35:13 DEBUG SaslRpcClient: tokens aren't supported for this protocol or user doesn't have one

Kerberos is working fine. But Spark didn't request a delegation token. I don't see a ozone delegation token returned.

Normally this happens when a file system is not in the default hadoop fs (fs.defaultFS). For a default FS, the MapReduce framework invokes FileSystem.addDelegationTokens() API to request its delegation token. Spark doesn't do this itself.

Your code may need to call FileSystem.addDelegationTokens() explicitly to make sure ozone dt is returned.

1 reply

MerelyOneMax1986 Aug 15, 2023
Author

@jojochuang thank you for diving so deep in the logs.
Exactly, it's the most meaningful line in the log. It's weird that in HDFS case Spark receives tokens easily.
So, I am using PySpark and do you know how to call analogue of FileSystem.addDelegationTokens() in PySpark?

And what do you mean by 'a file system is not in the default hadoop fs' ? I have this property in core-ste.xml

<property>
  <name>fs.defaultFS</name>
  <value>ofs://om-0.om.default.svc.realm/</value>
</property>

Thanks in advance!

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Ozone doesn't work with Kerberos and Spark #4996

{{title}}

Replies: 3 comments 8 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

Ozone doesn't work with Kerberos and Spark #4996

MerelyOneMax1986 Jun 28, 2023

Replies: 3 comments · 8 replies

MerelyOneMax1986 Jun 28, 2023 Author

MerelyOneMax1986 Jun 28, 2023 Author

MerelyOneMax1986 Jun 29, 2023 Author

cloudcheflabs Jul 1, 2023

MerelyOneMax1986 Jul 2, 2023 Author

cloudcheflabs Jul 2, 2023

ozone-site.xml

core-site.xml

MerelyOneMax1986 Jul 3, 2023 Author

cloudcheflabs Jul 4, 2023

MerelyOneMax1986 Jul 4, 2023 Author

jojochuang Aug 7, 2023 Collaborator

MerelyOneMax1986 Aug 15, 2023 Author

MerelyOneMax1986
Jun 28, 2023

Replies: 3 comments 8 replies

MerelyOneMax1986
Jun 28, 2023
Author

MerelyOneMax1986 Jun 28, 2023
Author

MerelyOneMax1986 Jun 29, 2023
Author

cloudcheflabs
Jul 1, 2023

MerelyOneMax1986 Jul 2, 2023
Author

MerelyOneMax1986 Jul 3, 2023
Author

MerelyOneMax1986 Jul 4, 2023
Author

jojochuang
Aug 7, 2023
Collaborator

MerelyOneMax1986 Aug 15, 2023
Author