Local preview freezing for ~1sec (TextureRenderView only) when starting the meeting in the background

[tmilovan123]

Describe the bug
If you show a local preview (as described here) using TextureRenderView and call audioVideo.start() while the preview is still showing, you would see frames freezing (dropping?) on the preview visibly for ~1sec.

Context

One of the use cases in our app is to display local preview (state 1) before the call starts and animate it into the proper (tile-based) local video views (state 2). To make sure both local and remote video views display something before transitioning to state 2 (we don't want to show black screens), we start the meeting while the local preview is still shown and wait until some frames get rendered onto local and remote video views, then transition to state 2.

To Reproduce
Steps to reproduce the behavior:

1. Have a local preview of type TextureRenderView backed by DefaultCameraCaptureSource,
2. While the preview is displaying and rendering video, call

audioVideo.start()
audioVideo.startRemoteVideo()
audioVideo.startLocalVideo(cameraCaptureSource) // Same one as in use by preview

You will see the video freezing temporarily on the preview and continuing smoothly afterwards.
You can modify the Chime demo app to reproduce this issue. I have attached a diff with the simplest issue repro (starting a meeting in DeviceManagementFragment with 3 sec delay after preview start).
Repro.patch.txt

Note: this does not reproduce with SurfaceRenderView. Currently using DefaultVideoRenderView as a workaround for preview in our app.

Logs
logs.txt

Test environment Info (please complete the following information):

• Device: any
• OS: any (mostly ran with Android 10/11)
• Version amazon-chime-sdk: 0.11.2
• Version amazon-chime-sdk-media: 0.11.2

Additional context
I have tried executing audioVideo.start() on another thread (simply by using CoroutineScope(Dispatchers.IO).launch{...}. It makes it better for the demo app (even if I make the preview full-screen), but the issue is still visibly present in our app even with this workaround. Also, I'm not sure this method is thread safe.

I also investigated a bit which code causes this drop, and it seems so far that audioClientController.start(...) (see DefaultAudioVideoController#start(...)) is the root cause, specifically some code in DefaultAudioClientController#start(...). Not sure what the connection is here. This guess is only based on random experiments with moving some of the audio code to another thread and making the issue to be less severe. However, this could totally be a false conclusion, I did not look thoroughly enough to make it definitive, nor did I profile these calls. Also, again, the effect of moving different code to another thread was not that pronounced in our app vs demo app, so it makes me think that it is either a false cause or the audio native code is causing it somehow.

[hokyungh]

Thank you for detailed explanation and reproduce step. We'll investigate and get back to you.

[tmilovan123]

Correction: we discovered some issues with using DefaultVideoRenderView - it seems to create artifacts in our on-top animations when the device is in use extensively. So we will need to drop it as a workaround :(

[hokyungh]

@tmilovan123 I think your conjecture is correct. The problem seems to be in code where we are using main thread to start the audio client, which takes over UI thread causing some freezing issue there.
We'll try to see if we can move this to other thread and not cause any issue.

[tmilovan123]

@hokyungh Thanks for getting back! One thing, can you ping me when you have a potential fix so I could test with a dev version of the SDK in our app? The reason is, I tried moving it to another thread by myself, building an SDK and using in our app, and it still had some effects of freezing, even though it appeared fixed in demo app. So I would like to get to the bottom of it as well. Not sure if we do something special there to make the situation worse. Thanks in advance.

[tmilovan123]

@hokyungh As a result of our conversation, I can confirm that your fix + moving DeviceController#chooseAudioDevice(…) to be called on Dispatchers.Default resolves the issue. My question therefore is, do you see any potential issues with calling chooseAudioDevice(...) on another thread, or is it safe to do?

[alnlau]

Hello @tmilovan123,

The only potential issue I currently see is if listAudioDevices() and chooseAudioDevice(...) was called at the same time on an Android device below API level 23 since both use AudioManager (specifically isBluetoothScoOn and setBluetoothScoOn). Upon audio devices changing, DefaultDeviceController will call listAudioDevices() on the main thread for onAudioDeviceChanged(...). We'd have to do further investigation on our usage of AudioManager.

Other than that, I currently don't see any potential issue with calling chooseAudioDevice(...) on a different thread provided that it and audioVideo.start() are both called on that same thread.

[tmilovan123]

Hi @alnlau ,

Thanks for your response. Our minSdkVersion is 24, so it is not a risk for us. It's good to know AudioManager is thread-safe in newer versions of Android.

Based on this, we can move both calls to another thread, and this should resolve/mitigate the impact. I know that part of the audioVideo.start() code uses uiScope (not sure how much it contributes overall), I think @hokyungh's CR will be the last change that fully completes the fix for the issue. Thanks!

[dylonChime]

I'll close this for now. Feel free to reopen if you have further questions. We have an open issue for documentation: #267