New EMS collector clarification

[wally007]

Hello,

thank you for implementing EMS collector which we hope will fix issues with our health monitoring we had few months ago
#885

After reading the documentation for EMS collector - I would like to get clarification on how we can get overall health of the cluster as far as EMS collector is concerned ?

From what I read, we would have to write, collect and alert on 3k metrics if we wanted to replicate "Health" like is in the System Manager, is that correct ?

Is there anyway for harvest to just give us an overall EMS health ?
Or maybe change the behavior of the existing metric "cluster_new_status" to include EMS alerts ?
Or introduce a new metric that would provide overall EMS health ?

Thank you

[rahulguptajss]

@wally007 As per system manager network calls, it does invoke below Rest calls to decide on these health alerts. We could map them to EMS or write a plugin/template which provides the same.

https://CLUSTER_IP/api/storage/disks?fields=name%2Ccontainer_type%2Cnode&return_timeout=120&container_type=broken%7Cunassigned
https://CLUSTER_IP/api/storage/shelves?local=true&fields=name%2Cid%2Cmodel%2Cmodule_type%2Cserial_number%2Cinternal%2Clocal%2Cstate%2Cdisk_count%2Cbays%2Cstate%2Cpaths%2Cfrus.firmware_version%2Cfrus.type%2Cerrors&return_timeout=120
https://CLUSTER_IP/api/cluster/nodes?fields=controller.frus%2Cname%2Cmodel%2Cha%2Cserial_number%2Cstate%2Cmetrocluster%2Cis_all_flash_optimized&return_timeout=120
https://CLUSTER_IP/api/cluster/chassis?fields=nodes.position%2Cnodes.name&return_timeout=120
https://CLUSTER_IP/api/private/support/alerts?return_records=false&return_timeout=120&suppress=false&time=%3E%3D24h
https://CLUSTER_IP/api/network/ethernet/ports?return_records=false&return_timeout=120&enabled=true&state=down
https://CLUSTER_IP/api/network/fc/ports?return_records=false&return_timeout=120&enabled=true&state=offlined_by_system
https://CLUSTER_IP/api/storage/volumes?fields=uuid%2Cname%2Cmovement.state%2Csvm&movement.state=cutover_wait%7Cfailed%7Ccutover_pending&return_timeout=120
https://CLUSTER_IP/api/network/ip/interfaces?return_records=false&return_timeout=120&location.is_home=false
https://CLUSTER_IP/api/support/ems/destinations?return_records=false&return_timeout=120
https://CLUSTER_IP/api/storage/volumes?fields=uuid%2Cname%2Cmovement.state%2Csvm&movement.state=cutover_wait%7Cfailed%7Ccutover_pending&return_timeout=120
https://CLUSTER_IP/api/network/ip/interfaces?return_records=false&return_timeout=120&location.is_home=false
https://CLUSTER_IP/api/storage/volumes?fields=state&return_records=false&return_timeout=120&anti_ransomware.state=enabled&anti_ransomware.attack_probability=low%7Cmoderate%7Chigh

[wally007]

It'd be great if it was added to EMS - but looking at the list you've provided - it doesnt seem to be the full list of what System Manager uses.

In my previous issue when I reported that Harvest does not include EMS alerts in its health reporting
#885
I can see that the SystemManager basically alerted me to miss-configured LDAP server configuration.

This event doesnt seem to be among the events you've listed above.

[cgrinds]

hi @wally007 #885 is from System Manager's Events tab, which sends this ONTAP REST request
https://10.193.48.11/api/support/ems/events?return_timeout=120&max_records=20&fields=time,node,message,source,log_message,index&order_by=message.severity&offset=40&message.severity=alert|error|emergency

That's the same endpoint that Harvest is using with its per-event style EMS collector

Harvest's EMS template lets you pick specific EMS events and alert on those. We shipped with 60 high value EMS events, but I understand that doesn't fit your need.

Sounds like you'd like an EMS based cluster health roll-up. We'll take a look at adding that. Looks straightforward to build something similar to System Manager. SM is asking for the last 20 or 40 EMS events that have a severity of alert|error|emergency, in addition, to some other number of REST requests per Rahul's earlier message. When EMS events are returned to SM, it groups by severity, and updates the UI.

Harvest can do something similar and publish a cluster_health or cluster_health_status. If there are zero events of severity alert|error|emergency the value would be 0, otherwise 1, 2, 3, etc.

Does that capture your request?

[wally007]

Hi @cgrinds ,

Harvest's EMS template lets you pick specific EMS events and alert on those. We shipped with 60 high value EMS events, but I understand that doesn't fit your need.

-> Correct. While it is great that one can cherry pick each EMS event separately, we just want to know overall cluster health status in Grafana and alert on it.

Sounds like you'd like an EMS based cluster health roll-up. We'll take a look at adding that. Looks straightforward to build something similar to System Manager. SM is asking for the last 20 or 40 EMS events that have a severity of alert|error|emergency...

-> This would be great.

Harvest can do something similar and publish a cluster_health or cluster_health_status. If there are zero events of severity alert|error|emergency the value would be 0, otherwise 1, 2, 3, etc.

Does that capture your request?

-> Yes, this would be great (ability to get alert|error|emergency separately would be great) but it's okay if its grouped into a single metric.

Thank you !