Take steps to avoid threadpool starvation

[Erarndt]

Fixes #

Context

There are a handful of areas where threadpool threads are unnecessarily blocked by synchronous work, or there is additional contention that can be avoided. The most egregious instance is the synchronous read in BuferedReadStream

The outer call to BeginRead() ends up calling BufferedReadStream.Read() which synchronously blocks when _innerStream.Read() is called. This results in a substantial amount of block time for threadpool threads.

Changes Made

Testing

Notes

[JanKrivanek]

Related to #11160

[JanKrivanek]

/azp run

[azure-pipelines]

Pull request contains merge conflicts.

[SimaTian]

Is there any advantage for a ConcurrentQueue as opposed to using Channel?
I'm not saying that we should use channel - when I was researching this piece of code, I've stumbled onto it and it piqued my interest. Hence the question.

[SimaTian]

Isn't it time to consider having a separate thread for the drain? with the "new" await, it should not even require a dedicated thread. (unless I'm grossly misunderstanding something)

Building on the channel idea from previous comment:
We could set up a multi writer, one reader channel.
Then have one thread to drain the channel
e.g. while (await channelReader.WaitToReadAsync()) {doTheDraining()}

[SimaTian]

Aside from that, the only place that is ever doing the Draining is this one:

 public void SendData(INodePacket packet)
 {
     if (IsExitPacket(packet))
     {
         _exitPacketState = ExitPacketState.ExitPacketQueued;
     }
     _packetWriteQueue.Add(packet);
     DrainPacketQueue();
 }

it is also the only place that ever places packets into the Queue.

since we're already touching this place, I would like to take it further if possible.