Replication Type RATIS, STAND_ALONE, CHAINED and EC.

[mosleyit]

We have deployed ozone in a 3 node HA Cluster.

We have found that if a node goes down then writes to the s3 gateway will fail. Our assumption is that this is due to the strong consistency requirement of Ratis.

Our question is if we change

  <property>
    <name>ozone.server.default.replication.type</name>
    <value>RATIS</value>
    <tag>OZONE</tag>
    <description>
      Default replication type to be used while writing key into ozone. The
      value can be specified when writing the key, default is used when
      nothing is specified when creating key or no default value set at bucket.
      Supported values: RATIS, STAND_ALONE, CHAINED and EC.
    </description>
  </property>

Changing this to CHAINED does this just change to eventual consistency?
Are there any other settings where we can set the minimum replication prior to a write confirmation?
Are there any drawbacks to this?
Will it perform as expected in that if a node goes offline or two we will continue to be able to write and read?

Since we only have three nodes we do not want the cluster to effectively fall down if we loose a node.

[adoroszlai]

• Please don't use CHAINED, it was never implemented. OM will fail to start with that setting. The reference in config description is unfortunately left over (HDDS-12144). Thanks for pointing this out.
• Putting metadata and data on the same nodes is not recommended.
• Write may succeed to two datanodes if the third datanode goes down while writing the last block. Further writes will not work (Unable to allocate a container ... Unable to find enough nodes ... Required 3. Found 2.).
• Read works as long as you have at least one replica (with 3 datanodes: at least one of the three) available.

[mosleyit]

Okay thank you for that. We are currently running RATIS with OM and SCM running in HA.

We have three nodes. Each of the nodes is running OM SCM and DATANODE.

We have found that when 1 node goes down we can no longer PUT or GET files. We can perform list functions.

Is this the expected behavior?

If this is then the documentation should be updated that in a production cluster that it is recommended to have 5 nodes vs 3 as 4 nodes could cause split brain.

[errose28]

We have found that when 1 node goes down we can no longer PUT or GET files. We can perform list functions.

This implies that the issue is on the datanode side, since they are needed for put and get (data ops) but not list (metadata ops). If you are writing 3-way replicated data PUT will fail with only two datanodes. Ozone will not let you commit data with less than your specified durability.

GET of already persisted data should work with 2/3 datanodes available though. Can you share the error you are seeing when this happens?

If this is then the documentation should be updated that in a production cluster that it is recommended to have 5 nodes vs 3 as 4 nodes could cause split brain.

I'm not following exactly what you mean with regards to split brain here, but it is important to note that Ozone has different components that are meant to be deployed independently. For example for high availability you should have 3 OMs, 3 SCMs, and at least 3 datanodes, but ideally more to handle failures.

Just to clarify, when you say 4 node cluster, that should not be 4 nodes each running an OM, SCM and datanode. OM and SCM use Raft for consistency which needs an odd number of nodes. Currently the only supported configurations for OM and SCM groups are 1 node or 3 nodes. Any number of datanodes is supported as long as you have enough to write data with the redundancy specified.