NiFi Disaster Recover

[deebify]

Can you setup a NiFi disaster recovery by setting up to NiFi cluster C1, and NiFi C2 with the same configuration. and in case of any cluster failure I can switch to the second cluster with synced flow?

[maltesander]

Hi @deebify,

this would require some mechanism to stop and disconnect a node in C1, do the same for all nodes in C2, copy the flow.gz from C1 to C2 and restart all nodes in C2 (which can take a long time depending on the flow.gz size).

If you want to switch over in case of failure in C1, your flow in C2 is very likely outdated and/or the cluster is not running anyways due to the flow sync.

Another problem would be which cluster is running the flows, primary node processors etc.

I dont see a point in this and NiFi is pretty failure tolerant by itself.

Since this is more of a NiFi than Stackable question i would suggest to ask on the NiFi mailing list for advice for this topic.

Malte

[deebify]

For my case there's an environment in data center C1 which already synced with environment in C2 resources. So all PVCs are synced even the flow.json.gz and flow.xml.gz in databaseRepo.

The mechanism you described—disconnecting nodes, transferring the flow.gz—is a form of moving from cluster to another cluster but this is not exact my case since my DC2 is already synced.

From an idea perspective, if the C1 cluster is operational, then in the event of a failure in DC1, I will manually intervene to start the NiFi deployment in DC2 using the synchronized PVCs. Once DC1 is up again, I will do the opposite.

I am just discussing the scenario: Is there something that will not make it work?

[maltesander]

I think if you got the sync jobs up and running and they are working properly, you should be fine.

I will manually intervene to start the NiFi deployment in DC2 using the synchronized PVCs

If this means actually starting up the NiFi cluster/nodes, you definitely will have a downtime depending on how much time it takes the nodes to come back and sync.

Would you mind to elaborate how you sync the PVCs?

[deebify]

Indeed, there will be a downtime. I’m fine with some downtime for manual steps, though—it’s nothing compared to a full primary DC outage.

check out volsync.