transport: better handling for persistent streams on preferred network change

[rebello95]

Problem

Envoy Mobile currently listens to network changes (i.e., wifi to cellular) and switches preferred_network based on these notifications. This ensures that future requests/streams made through Envoy Mobile are performed on the preferred/current network.

This approach deteriorates in its current form when applied to long-lived persistent HTTP streams, as these streams may silently "die" as the underlying connection on the non-preferred network is shut down.

The issue manifests itself more commonly in situations like this one:

• User starts a persistent stream on wifi
• User enters an elevator, changes floors / leaves the building, and goes out of range of the wifi network
• User's phone switches to cellular
• Envoy Mobile switches future requests to use cellular
• Initial stream hangs until it times out

Options

Some things we could do to improve this:

1. (Short term stop-gap) Immediately shut down persistent connections when the network changes
2. Tune timeouts for persistent connections so that they are shut down on their own when a connection dies based on timeout
3. Potentially use separate clusters with different configurations for long-lived streams. Note that this would require us to identify which connections are "long lived" when they're started

Ideally this issue should be fixed within Envoy Mobile so that we can prevent making end consumers implement fixes for this on their own, and so that consumers get additional fixes for free as the library evolves in this area.

Related: #13