2024 Feb APAC Community Sync

[ChenSammi]

Feb 2th
Attenders: Yiyang, Xi(Pony), Hao, Minyu, Bin, Hualong, Yuanben, Jianghua, Conway, Jeff, Wei-Chiu, Sammi

Cloudera:

• Share Ozone append initial design.
• Present the web site under development.
• Suggest community to pay attention to the "Ozone Storage Container Reconciliation" design and better to have a review.

DiDi:

• Suggest one improvement on Recon UI, fold the pipeline info for a DN on Datanode page. When a Datanode has more than 10 pipelines, current it will display all the pipelines by default, which is not necessary most of the time. @devmadhuu
• OM transaction ID crash bug fixed in HDDS-9876. One more further step to help to recover the OM DB, would be a CLI tool to update the transaction id in RocksDB. Filed HDDS-10295
• Testing Follower reader performance. One observation about Follower read, the client always read from the the first SCM defined in the HA group. It's better to randomly select some SCM to start with, so the read requests will be served evenly across SCM instances. @szetszwo .
• Reported one block missing issue. After a further investigation after the meeting, it is caused by some customized lock changes in DN side. After reverting the changes, the issue should be fixed.

Shopee:

• Discussed unsupport HCFS APIs in Ozone
  • append, truncate
  • access time
  • EC policy change API
  • no directory rename for open for writing files.
• Discussed the way to share files between different users.
  • S3 has such capability called presigned URLs. Ozone should have supported this too.
  • symbolic link is not supported in Ozone so far. And there is no plan to support it yet.

Qihoo:

• Find one performance issue with volume check, if disk is not physically healthy, the volume check process which try to write and read a small mount of data file will itself hang for around 1 hour, then the disk cannot be marked as failed timely. During this 1h, may read/write operations scheduled on this disk, causing the whole DN read/write latency extremely high. The solution is found and will submit patch later.
• DELETED state containers in memory. There are many DELETED state container info held in memory after a bunch of data writes, then deleted shortly after that. This data in memory will never be freed, nor does it be deleted from SCM rocksDB table. Propose a lazy way to delete these containers from memory and rocksDB. Will have a simple design document and share with the community.
• Block location cache, which is not cleared and so the client always fetches from the same DN for the same block. (XceiverClientGrpc.getBlockDNcache). This will be a problem for long running services like S3 gateway, which usually holds the client for its entire life. So some DNs will become read hot spot.

[jojochuang]

Tagging @duongkame @kerneltime regarding the Block location cache issue. I think it makes sense.

[kerneltime]

This goes in contrast to the use case that the client should always from the closest Datanode for a given block based on topology. If the Datanode is overloaded the retry logic does round robin. Maybe the logic to sort Datanodes should move to the client where the client can decide to randomize if there is no locality to be gained.

[szetszwo]

... recover the OM DB, would be a CLI tool to update the transaction id in RocksDB. Filed HDDS-10295

We may use the rocksdb_ldb tool (provided by Rocksdb) for the moment:

$ rocksdb_ldb --db=/var/lib/hadoop-ozone/om/data/om.db/ scan --column_family=transactionInfoTable
#TRANSACTIONINFO ==> 11#282615
$rocksdb_ldb --db=/var/lib/hadoop-ozone/om/data/om.db/ put "#TRANSACTIONINFO" "11#573145" --column_family=transactionInfoTable
OK
$rocksdb_ldb --db=/var/lib/hadoop-ozone/om/data/om.db/ scan --column_family=transactionInfoTable
#TRANSACTIONINFO ==> 11#573145

See also https://github.com/facebook/rocksdb/wiki/RocksDB-Troubleshooting-Guide

... One observation about Follower read, the client always read from the the first SCM defined in the HA group. It's better to random select ...

I guess you mean OM instead of SCM? Randomization sounds good. Clients may also choose the closest OM to read from.

Thanks everyone for the discussion!

[dombizita]

Suggest one improvement on Recon UI, fold the pipeline info for a DN on Datanode page. When a Datanode has more than 10 pipelines, current it will display all the pipelines by default, which is not necessary most of the time.

Thanks for the suggestion, it makes sense, @devmadhuu filed HDDS-10301 for it.

Is the Ozone append initial design available somewhere?

[errose28]

There are many DELETED state container info held in memory after a bunch of data writes, then deleted shortly after that. This data in memory will never be freed, nor does it be deleted from SCM rocksDB table.

I agree we should not hold deleted containers in memory. I don't think we should be too concerned about deleting them from RocksDB right now. There are many corner cases around doing this and SCM is far from maxing out its RocksDB capacity. There was lots of discussion around this in #3360 a while ago.

[kerneltime]

Thanks for bringing this up #6121 will need more eyes to close the PR and proceed with implementation.