MediaStreamTracks are ended after returning from sleep

[a-type]

• I have verified that the issue occurs with the latest twilio-video.js release and is not marked as a known issue in the CHANGELOG.md.
• I reviewed the Common Issues and open GitHub issues and verified that this report represents a potentially new issue.
• I verified that the Quickstart application works in my environment.
• I am not sharing any Personally Identifiable Information (PII)
  or sensitive account information (API keys, credentials, etc.) when reporting this issue.

Code to reproduce the issue:

• Run the latest video quickstart example
• If relevant, I'm on MacOS 11.2.3 using a Macbook Pro 13-inch with Core i5 (2020)
• Join with 2 participants, both sharing audio/video
• Put the computer to sleep for any length of time
• Restore the computer from sleep

Expected behavior:

• When Twilio reconnects after device sleep, either:
  • Track publications are restored and reconnected
  • OR, track publications fail and are cancelled or otherwise emit errors

Actual behavior:

The room reconnects after sleep and track publications are re-established, but the underlying MediaStreamTrack objects for remote tracks have readyState: 'ended' and no media is transmitted from remote participants. Each participant sees their own media stream as active, but peers' media streams as empty (black video).

room.state is connected.

Here is a full log from one of the peers. Included are some expression evaluations which indicate that the remote video publication has isSubscribed: true, isTrackEnabled: true, the track has isEnabled: true, and the MediaStreamTrack has readyState: ended. However, when inspecting the corresponding local stream track (which the other peer client sees as readyState: ended), the readyState is live (as seen on the last line).

No error events are emitted. From a client perspective it appears as if the room reconnection was successful and all local tracks are published and streaming, but all peers receive empty tracks and the client cannot see or hear anyone else despite receiving their publications.

If new publications or new peers join, we see a new behavior - the peers are shown, and remote track publications are added, but they are never subscribed and .track stays null. When a new peer joins a client which was sleeping, the sleeping client's tracks are likewise unsubscribed and never subscribe. Both clients show their own local tracks as published and active.

So, to summarize:

• Reconnecting after sleep, all remote media tracks are black and in ended state.
• All local tracks appear to behave normally.
• If new peers join the room after your client has been sleeping, they will show up but their tracks will never subscribe. Your tracks will never subscribe on their clients, either.

Software versions:

• Browser(s): Chrome 89.0.4389.114
• Operating System: MacOS 11.2.3
• twilio-video.js: 2.13.1
• Third-party libraries (e.g., Angular, React, etc.):

If anyone can find a workaround or way to detect this bad state I'd be fine with that. The browser does not give any indication of power state changes that I'm aware of, and I can't use the page visibility API as I don't want to assume a failure if the user just switches tabs or minimizes the app.

I've tested reconnections in other scenarios, including loss of internet (via disabling the device or poor connection), and they don't reproduce this behavior. It seems consistently tied to cycling sleep. While this may seem like an edge case, our app has long-lived sessions and we regularly see users sleep their computers while a tab is open. In these cases users may return to the tab later and be confused when A/V is (silently) not working or may not even notice the problem until later.

[a-type]

Because this issue is a little hard to describe well I've recorded videos of it.

https://youtu.be/qKrDZfiS1xo

The video shows the full behavior, starting with connecting two clients in the demo app, then putting the computer to sleep and waking it. I then walk through the two behaviors I see: one for clients which come back from sleep, and one when new clients connect to them and try to share tracks.

[makarandp0]

Hello @a-type, Thank you for the detailed description and the video. I was able to repro this issue. On chrome I notice that track goes to the "ended" state very quickly after sleep. It stays in "live" state in both firefox and safari if the computer was quickly woken from the sleep.

I have filed a bug for chrome.

One workaround that comes to mind is to watch for RemoteVideoTrack going into "ended" state and then communicate to publisher using data channels that the track was "ended", and they need to unpublish and republish.

Thanks,
Makarand

[a-type]

Good idea, thanks for narrowing it down. I'll see about coordinating that.

[makarandp0]

Hey @a-type, We looked bit more into this, and I think SDK should be able to detect this case and should disconnect the publisher in this case. I have filed a bug (VIDEO-4794) to track this issue.

In the meanwhile, I thought of a bit simpler workaround that could help in your scenario. On chrome we see that peer connection object goes to 'closed' state when the machine goes to sleep. You can detect such state change on publisher side and once detected disconnect the participant.

Note: This workaround requires access to some internal state of the room:

const Video = require('twilio-video');
const room = await Video.connect('$TOKEN', { name: 'room-name' });  
const pc = room?._signaling?._peerConnectionManager?._peerConnections.values()?.next()?.value?._peerConnection;
if (pc) {
  pc.addEventListener('signalingstatechange', () => {
    if (pc.signalingState === 'closed') {
      console.log('Disconnecting because PeerConnection was closed: ')
      room.disconnect();
    }
  });
}

Note: Peer connection can go into closed state for other reasons, but in all such cases disconnect is a correct thing to do anyways. Let me know if that helps.

Thanks,
Makarand

[a-type]

Oh, much easier! Thanks, I'll probably use that workaround and incorporate it into our extensions of the automated reconnection logic.