Migrating data from S3 to Ozone

[gigem]

Are there any tools for easily copying or mirroring data from S3 to Ozone? If POC testing goes well with Ozone, we will need to migrate several TBs of Hive data from S3 to Ozone. Mirroring (with incremental changes) would be best but simply copying directories one at atime in whole would also work.

[arp7]

Hi @gigem, what is the scale of data you are looking to move? Also are you looking at a one time data movement or incremental replication.

Distcp is well-tested and most widely used for data migration.

[gigem]

Hi @arp7. This would be a one time move. The catch is that some of the data is updated nightly.
Ideally, everything would be copied in bulk and then one or more incrmental copies done to pick up the changes while the bulk copying was being done. At that point, the update processes could be turned off temporarily and everything reconfigured to use the new location.
If incremental updates can't be done, copying in pieces would be the fallback method. It would work but would be a lot more tedious to keep track of what has adn hasn't been moved.