Clear pooled connection after node health checks

[merlimat]

When using Istio between coordinator and storage node, we should discard all the pooled connections in the coordinator, after a node is declared failed (and then it recovers).

The problem is that the gRPC client is not able to detect that TCP has failed, because of the Istio proxy. eg:

• the istio is still setup to talk with the old pod
• coordinator uses the pooled client and writes to a gRPC stream. There is no failure, even though the channel is not valid

In normal conditions, gRPC will process the TCP failure and open a new connection, though this does not work with Istio in between.

Modifications

Force to discard all the pooled gRPC clients for target node once we have successfully reconnected with the new pod.