HA not working in cloudstack 4.19

[Aashiqps]

I have created a cloudstack with 1 management node and 3 kvm host nodes. For primary storage i am using linstor in disaggregated method and configured as diskless as i am using separate nodes ( 1 combined and 2 satellite nodes ) for linstor cluster and also another node for nfs secondary storage. The cloudstack is working but i am facing challenge in HA and snapshots. The vm created are not migrating to another host when i reboot, remove network cable or stop cloudstack agent in KVM host nodes, also the snapshots are not working. However for maintenance mode in host the vm, systemvm and vr are automatically migrating to another host. can anybody guide a solution.

[shwstppr]

@Aashiqps is it possible for you to check behaviour with NFS primary storage?

[rp-]

VM snapshots is also a new feature in 4.19.1, before only disk snapshots worked.
But again there are also a lot of fixes for disaggregated setups in 4.19.1.
So I would wait for the RC1 or the release.

[slavkap]

By the way @Aashiqps, with the current version you're using you can check this option for the VM snapshots and Linstor:

Enable the global configuration setting kvm.vmstoragesnapshot.enabled
For a live snapshot of a VM instance on a KVM hypervisor without memory. Requires qemu version 1.6+ and qemu-guest-agent installed on the guest VM

[Aashiqps]

@slavkap i have tried these settings and snapshot not working.

@shwstppr @rp- I have checked VM HA and it is working with linstor. mentioning some of my observation and issues regarding the setup i have made.

The network communication is working only when the instance and the VR are in same KVM host, while creation sometimes VR might be created in another host and we have to manually migrate the VR to the same host where instance are created for the network to work.

All VM Instances in the Host, Virtual Routers and System VM (if present) are automatically migrated to another Host when Host Maintenance Mode is enabled but If the instance and its VR are migrating to separate hosts the network communication issue arises.

[shwstppr]

@Aashiqps network communication with different hosts might be due to some configuration issue. You may refer to some old reported issues such as #9027, #8360

[Aashiqps]

Thanks, enabling trunk mode in physical switch resolved the issue. Snapshots and Host HA are still a challenge, hope in next release it can be resolved.

[btzq]

Hey @Aashiqps , we are facing the same situation as well. Great to have found another Linstor user in dissagregated architecture.

At this moment, we managed to get Volume Snapshots to work fine, after latest fixes from Linbit side.

But we still the failover issues, where we cant seem to get all the Virtual Routers to start up successfully during a node failure (we simulated by pulling the power from the server). If the Virtual Router cant start up, then the VMs in that network cant continue to start up as well.

We tested this using NFS Too just to isolate the network issue and the VM HA using NFS works just fine.

When we go through the logs, we cant seem to identify whats the problem.

Our latest findings is that according to the logs, the VR was successfully migrated to the new host, and its status transitioned to running. However, the 'ACS HighAvailabilityManager' triggered a stop/reboot action on the router and did not take any action to start it afterward.

`2024-07-16 17:31:18,853 DEBUG [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-137:ctx-b555a5f3 job-385919/job-386245) (logid:1dc1e938) Run VM work job: com.cloud.vm.VmWorkStop for VM 54572, job origin: 385919
2024-07-16 17:31:18,854 DEBUG [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-137:ctx-b555a5f3 job-385919/job-386245 ctx-10b406cf) (logid:1dc1e938) Execute VM work job: com.cloud.vm.VmWorkStop{"cleanup":true,"userId":1,"accountId":1,"vmId":54572,"handlerName":"VirtualMachineManagerImpl"}
2024-07-16 17:31:18,867 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-137:ctx-b555a5f3 job-385919/job-386245 ctx-10b406cf) (logid:1dc1e938) VM instance {"id":54572,"instanceName":"r-54572-VM","type":"DomainRouter","uuid":"a0022aec-996e-490d-8a21-3eccd43c9e0b"} state transited from [Running] to [Stopping] with event [StopRequested]. VM's original host: Host {"id":129,"name":"n2ncloudmy1cp02","type":"Routing","uuid":"3bf16d9d-e561-4e59-b855-7256bee35c6f"}, new host: Host {"id":129,"name":"n2ncloudmy1cp02","type":"Routing","uuid":"3bf16d9d-e561-4e59-b855-7256bee35c6f"}, host before state transition: Host {"id":129,"name":"n2ncloudmy1cp02","type":"Routing","uuid":"3bf16d9d-e561-4e59-b855-7256bee35c6f"}

2024-07-16 17:31:37,947 INFO [c.c.h.HighAvailabilityManagerImpl] (HA-Worker-42:ctx-cfc0e084 work-103139) (logid:fe9b0f9a) VM
VM instance {"id":54572,"instanceName":"r-54572-VM","type":"DomainRouter","uuid":"a0022aec-996e-490d-8a21-3eccd43c9e0b"} is no
w no longer on host 129
2024-07-16 17:31:37,947 INFO [c.c.h.HighAvailabilityManagerImpl] (HA-Worker-42:ctx-cfc0e084 work-103139) (logid:fe9b0f9a) Completed work HAWork[103139-HA-54572-Stopped-Investigating]. Took 1/10 attempts.`

But the challenge we are having, is that we dont see any error related to the storage. There should be somewhere a message that either a device can't be accessed or created, but we cant find any.

But there are few things to take note when using Linstor:

• Theres no need to use IMPI OOB. Infact, users are asked not to. This is cause only NFS iSCSI storage are susceptible to splitbrain, but in Linstor, apparently the technology is different which is why splitbrain will not occur.
• Need to update Cloudstack Agent to not restart the server.
• HA Strategy in Cloudstack + Linstor is to rely solely on VM HA (not Host HA).

More info here:

https://linbit.com/drbd-user-guide/linstor-guide-1_0-en/#ch-cloudstack:~:text=video%20here.-,14.9.%20High%20Availability%20and%20LINSTOR%20Volumes%20in%20CloudStack,-The%20CloudStack%20documentation

https://linbit.com/drbd-user-guide/linstor-guide-1_0-en/#ch-cloudstack:~:text=14.9.1.%20Explanation%20and%20Reasoning

Im curious to know your progress and if you managed to find any solution to it? Happy to communicate to help each other out.

[Aashiqps]

@btzq Host HA is not working in our case, but VM HA is working. The instances, VR are automatically getting up when KVM host is recovered. VR dont migrate to other hosts if a power failure or network outage occurs (in my case). Check VM HA is working or not, by default it should work, just power off the instance ( not from UI ) and check instances are automatically starting or not. Reduced the time period of this parameter in global settings (kvm.ha.recover.wait.period). VM HA worked with both linstor and NFS primary

[btzq]

@Aashiqps so just to confirm your test results:

Scenario 1: Restart Hypervisor:

• Host HA = Does not work
• VM HA = Restarts In the same server after Hypervisor resumes

In both scenarios above, your results are expected. This is because a Hypervisor restart doesnt trigger any HA actually. All the services that were on the Hypervisor, will just restart automatically, on the same hypervisor.

This brings me to my second point, i think you are confusing VM HA. VM HA means that the VM will restart itself in another hypervisor. If your testing is only that the VM is restarted within the same hypervisor, then its not called VM HA. Its just only a server restart.

What is your definition of Host HA? My understanding is that you need OOB and IPMI for Host HA. But, in Linstor documentation, they explicitly requests not to use that as there is no need. https://linbit.com/drbd-user-guide/linstor-guide-1_0-en/#ch-cloudstack:~:text=video%20here.-,14.9.%20High%20Availability%20and%20LINSTOR%20Volumes%20in%20CloudStack,-The%20CloudStack%20documentation

Scenario 2: Power Off Hypervisor

• Host HA = Does not work
• VM HA = Does not work

In this scenario, the VM HA does not work (even for us), which is what we are trying to figure out.

[btzq]

@Aashiqps , also in our situation, we generally dont have problems when its VM thats restarting in another server during a node failure.

But when it comes to Virtual Router, we experience lots of failure rates.

Can you confirm if your testing is on VMs or VRs? And if VRs, how many did you test with? (We tested with 10)