rsync for live data cross-clusters

[atanasmatev]

Setup:

1. Two Openshift clusters with Openshift container storage on each (RWX, file system based default storage class)
2. Submariner used for cross-cluster network connectivity (Volsync exposed services are reachable between clusters)
3. Application is installed on the primary cluster that works with a huge database (100GB+) in a PVC

Question:
How to use Volsync properly to enable close-to-real-time replication of the data between the two clusters?
Is the "rsync" command configured to send only "deltas" of the PVC data as transferring 100GB+ on each iteration will be time consuming?
Is the only option to configure multiple PVCs in a namespace by creating a separate ReplicationSource/ReplicationDestination pair of objects for each PVC?

Thanks!

[JohnStrunk]

How to use Volsync properly to enable close-to-real-time replication of the data between the two clusters?
Is the "rsync" command configured to send only "deltas" of the PVC data as transferring 100GB+ on each iteration will be time consuming?

How close to "real-time" you can get is going to be a function of your data change rate and the storage system overheads. The more time it takes for the storage system to clone the source volume, the longer each iteration will take. Likewise, the more data that has changed, the longer it will take to send it over the network.

On each iteration, the source side clones the volume (assuming CopyMethod: Clone) to create a fixed point-in-time image to sync, then invokes an rsync pod against it. That rsync pod talks to another on the destination side to calculate the diff to transfer. Only the changed data will go over the wire, but all metadata must be compared, so there will be some overhead to that operation (dependent on file count and storage system performance).

If clones are slow on your storage system, if may take a while until the source rsync pod starts. This will directly affect the minimum iteration time you can use. One way around that would be to use CopyMethod: None on the source. This is an option since you're using RWX storage. This avoids the clone by invoking the source rsync mover directly against the app's PVC. The problem with this is that you lose the "point-in-time" consistency. There's no guarantee that what ends up on the destination is a consistent copy that ever existed on the source.

Is the only option to configure multiple PVCs in a namespace by creating a separate ReplicationSource/ReplicationDestination pair of objects for each PVC?

Yes. One ReplicationSource/ReplicationDestination pair per PVC. While we've considered approaches where a PVC could be annotated (#12) or grouped, we have not yet started work on it.

[themicster]

I too would love to have a way to do multiple PVCs per destination/src. I have tens of pods with 4x PVCs per pod, that's a lot of rsync destinations restarting every 5 minutes.

[tesshuflower]

I too would love to have a way to do multiple PVCs per destination/src. I have tens of pods with 4x PVCs per pod, that's a lot of rsync destinations restarting every 5 minutes.

We will hopefully be looking at making use of Volume Groups when it's implemented in the csi spec. Ideally this means we can make a VolumeSnapshot (volume group snapshot) that takes a point in time copy of multiple PVCs at once.

Some more info here: https://github.com/kubernetes/enhancements/blob/master/keps/sig-storage/3476-volume-group-snapshot/README.md