Twilio reports high packet loss on participant. But in GMeet it's fine.

[CyranMS]

Hi all,
For multiple weeks now, we have gotten a lot of angry customer complaints that the audio quality is horrible. Has anyone else noticed this? The audio quality is so bad that we can't continue the meetings. When we switch to Google Meet then, their network analysis reports packet loss of 0% on audio. Meanwhile, Twilio blames it on the participant's internet and packet loss. I noticed this now multiple times that Twilio reports >5% packet loss (sometimes around 10-20%), but it's fine on GMeet (packet loss <1%). Any reasons that could be the issue?

Cases:

1. RM1bace1ffac5418a834c373ebd35baff6
2. RM53702baf166107f0b5e5242a2d8725d0
3. RMd478e214d94e15745de4231da3df3059

[shadowfool]

We've had some similar complaints recently - as of around the last week of January.

[CyranMS]

Any updates? @manjeshbhargav @PikaJoyce @seancoleman2

[kNoAPP]

Hello, we have also been experiencing problems with audio, especially when joining the call. Users sometimes (like 5-10% of time time) can't be heard after joining. But can hear everyone else. These issues started occurring in the last couple weeks.

[manjeshbhargav]

Hi @CyranMS ,

Thanks for the report. I checked our internal logs for the Room SID you shared with me, and I did not see a lot of packet loss reported by our media server. The SDK however did not send any logs. Have you by any chance set insights: false in ConnectOptions? If so, can you set it to true and share some room sids where this is reproduced?

[CyranMS]

Thanks for getting back! The issue is that Twilio reports high packet loss and handles high packet loss way worse than Google Meet. Probably you have access to all our rooms anyway, so here is the link to the dashboard with it here. Where they use Google Meet in these cases then they do not get high packet loss communicated and the audio quality is way better. Are there any plans to improve this? @manjeshbhargav

[masterminh219]

Hey, can someone please share where I can get the Packet Loss from CLient side ? I dont want to access the dashboard but I want to integrate it to our system so our CS can reach out to them if needed

I used getStats but now it returns empty array

[vipulSoni458]

We are also getting the same high packet loss issues. Even though participant network quality is excellent and its passes all twilio network test but still there are lot of glitch in audio and video in session. Please look into this issue as its really impacting the customer experience and our business. Thank you

[CyranMS]

Any update @manjeshbhargav?

[CyranMS]

Any update and any plans to improve this?

[anna-vasilko]

Hi all,
So far our QE team was not able to reproduce described audio issues when testing in various environments including when simulating poor networks by introducing high packet loss or low up/down bandwidth. I am afraid there is not enough info to root cause the issue at this point.

Can any of you reproduce the issue consistently? If so, please provide exact reproduction steps, participant sids and browser logs for the impacted users.

[vbabenko]

@anna-vasilko Yes, we can reproduce it, try to perform POLQA tests, and compare it with Google Meet. Check the audio quality when you increase the package loss from 0-20%.
With increasing package loss you will notice significant problems with audio in Twilio, but Google Meet handles it on acceptable level

[spalmertwilio]

I am not seeing high packet loss when using the latest version and I am checking using our video insights in the Twilio console, the Room Monitor https://www.twilio.com/docs/video/troubleshooting/javascript-room-monitor and Chrome's built in WebRTC internals chrome://webrtc-internals/ even when simulating poor network conditions.

Whenever Media Quality is an issue we recommend reflecting on the app's use case and the settings used in the following areas:

• If the publisher (local user) is publishing at a large/high quality and whether the receiver (remote user) can easily handle the video stream from the publisher(s).
• If the remote user is using an old device, low powered device, or is using a low bandwidth network, the Network Bandwidth Profile API settings will help the receiver.
• Subscribe end-users to remote tracks efficiently.
• Simulcast (all three layers sent), Adaptive Simulcast (only layer(s) the remote users can receive and all are using will be sent), Simulcast disabled aka unicast.
• Sensible network settings, don't pin to media region when participants are located somewhere else. Use gll.
• Use diagnostic tools to pinpoint signaling and media connectivity issues and make them available to your end users, please see the diagnostic section of Twilio Docs to look for Video Insights, Room Monitor, Video Diagnostic Application.

Quality Improvement Recommendations Checklist For Group Rooms (Not for Peer-to-Peer or WebRTC Go Rooms)

• Use the latest SDK release.
• End users should be using either the Preflight https://www.twilio.com/docs/video/troubleshooting/preflight-api or Video Diagnostics App https://www.twilio.com/docs/video/troubleshooting/preflight-api#video-diagnostics-application prior to joining room to ensure the meet the networking requirements.
• Is Krisp enabled? In the [2.23.0 changelog] https://github.com/twilio/twilio-video.js/blob/master/CHANGELOG.md#2230-july-28-2022, Twilio Video released Krisp Noise Cancellation, a custom audio plugin utilizing Krisp's audio processing solution called NoiseCancellationAudioDevice.)
• Do not pin to specific regions instead use gll aka global low latency https://www.twilio.com/docs/video/tutorials/video-regions-and-global-low-latency
• Connect Options set to use adaptive simulcast https://www.twilio.com/docs/video/tutorials/working-with-vp8-simulcast#adaptive-simulcast.
• The camera and the screen share video source for the publisher are set to something reasonable in terms of render dimensions and framerate. Here's the minimum bandwidth recommendations https://www.twilio.com/docs/video/tutorials/developing-high-quality-video-applications#min-bw-recommendations but if the room has video quality issues they should not be publishing anything over 1080p if their end users are experiencing heavily degraded quality with low bandwidth, recommend they bump it down to 720p.
• Set Network Bandwidth Options to 'auto' for both contentPreferencesMode https://www.twilio.com/docs/video/tutorials/using-bandwidth-profile-api#understanding-contentPreferencesMode and clientTrackSwitchOffControl https://www.twilio.com/docs/video/tutorials/using-bandwidth-profile-api#understanding-clientTrackSwitchOffControl, and use maxSubscriptionBitrate https://www.twilio.com/docs/video/tutorials/using-bandwidth-profile-api#understanding-maxsubscriptionbitrate for end user on low bandwidth networks
• Developers should be following the guidance from the reconnection guide https://www.twilio.com/docs/video/reconnection-states-and-events and the best practices guide https://www.twilio.com/docs/video/build-js-video-application-recommendations-and-best-practices and I strongly recommend implementing the media warnings API.
• Finally, how many video tracks are attempted to being displayed on the mobile UI and the desktop UI? Was the appropriate Network Bandwidth Mode https://www.twilio.com/docs/video/tutorials/using-bandwidth-profile-api#understanding-mode (Collaboration, Presentation, Grid) and trackSwitchOff https://www.twilio.com/docs/video/tutorials/using-bandwidth-profile-api#track-switch-offs (detected, predicted, disabled) selected for this particular use case.

I also recommend reviewing

• https://www.twilio.com/docs/video/tutorials/developing-high-quality-video-applications
• https://support.twilio.com/hc/en-us/articles/13364709610011-Troubleshooting-Javascript-Video-SDK

For those who can reproduce the issue consistently, will you please check for packet loss in one of the Twilio tools or in the Browser tools? If it is present on one of those tools, please submit a support ticket and include the exact reproduction steps, participant sids, and browser logs for the impacted users.

[vbabenko]

Hi @spalmertwilio @anna-vasilko
Thank you for your detailed answer and all these useful links.
But we use Twilio for a really long time and we went through architectural review, and optimizations and have implemented all those mentioned APIs and we passed due diligence. We adjust bandwidth usage based on Preflight tests, Network Quality APIs, etc.
Context:
We use Adaptive Simulcast for 3+ participants, in most cases we use 640x480p@24fps per video. Grid view (max 6 participants), network options are "auto", GLL, TrackSwitchOff is based on network quality, etc.

I've submitted multiple support tickets but the answer is always the same...
However, Twilio's performance is dependent on the quality of network conditions. The issue in this case lies in how media servers handle package losses

Could you please run some professional tests to simulate real network issues such as package order variations, bitrate spikes, controlled package losses, delays, low receiver/sender networks, etc? These tests will help you identify any problems and drops in audio quality. The chrome dev tool is not the appropriate tool for these tests.

Thank you,

[spalmertwilio]

Thank you @vbabenko for the response. Do you mind sharing one of your support ticket numbers with me so that I may investigate?