Adding backward audio channel to your camera

Overview

This is an example of a possible workflow when adding audio backward channel support to a camera with a speaker to your Swift application. RTMP(S) is used as the data transfer protocol

To get the URL to which we will send our audio data, you need to use the getBackwardUrl method on CloudPlayerSDK object or a ready-made RTMP(S) URL obtained through the corresponding APIs: /cameras/{CAMID}/live_urls/

Required

1. You must have CloudSDK imported and everything related to it
2. The camera must support backward audio channel (NELLY type), which can be found through the corresponding API: /cameras/{CAMID}/audio/

    {
        "caps": {
            ...
            "backward_formats": [
            ...
            "NELLY",
            ...
            ],
            ...
        },
        ...
    }

Set up dependencies

1. Add MediaCaptureSDK.xcframework to your project
2. Include MediaCaptureSDK.h in your birdge.h file

#import <MediaCaptureSDK/MediaCaptureSDK.h>

3. Add the necessary permissions to use the microphone to your info.plist file

Implementation

1. Add MediaRecorder object and MediaCaptureConfig

    private let audioCapture: MediaRecorder = MediaRecorder()
    private let audioCaptureConfig: MediaCaptureConfig = MediaCaptureConfig()

2. Implement MediaCaptureCallback for monitor obejcts states

    extension StreamView: MediaCaptureCallback {
        func status(_ who: String!, _ arg: Int32) -> Int32 {
            if (arg == CaptureNotifyCodes.MUXRTMP_STARTED.rawValue || arg == CaptureNotifyCodes.MUXRTMP_STOPED.rawValue) {
                isStreamingStarted = (arg == CaptureNotifyCodes.MUXRTMP_STARTED.rawValue);
            }
            return 0;
        }
    }

3. Configure your MediaRecorder for backward audio only and open it

    func setupAudioCapture() {
        audioCaptureConfig.setStreamType(StreamType.STREAM_TYPE_RTMP_PUBLISH);
        audioCaptureConfig.setAudioFormat(AudioFormat.AUDIO_FORMAT_ALAW);
        audioCaptureConfig.setVideoFormat(VideoFormat.VIDEO_FORMAT_NONE);
        
        audioCapture.open(audioCaptureConfig, callback: self)
    }

4. Start backward audio channel. You must start the audio backward channel only after receiving the CaptureNotifyCodes.CAPTURE_AUDIO_STARTED notification

    func startStreaming() {
        guard !isStreamingStarted, let backwardUrl = cloudPlayer?.getBackwardUrl() else {
            return
        }
        audioCaptureConfig.setRTMPurl(backwardUrl)
        audioCapture.startStreaming();
    }

5. Starting audio data transfer.

    func micTouchDown(_ button: UIButton) {
        cloudPlayer?.mute(true)
        audioCapture.muteMicrophone(false)
    }

6. Stop audio data transfer.

    func micTouchCancel(_ button: UIButton) {
        cloudPlayer?.mute(isMute)
        audioCapture.muteMicrophone(true)
    }

7. Close backward audio channel.

    func stopStreaming() {
        audioCapture.stopStreaming()
    }

8. Close media recorder object and free memory.

    func stopStreaming() {
        audioCapture.close()
    }

Additional features

1. Additional processing of input audio data. If you want to further modify the data received from the microphone, you can take advantage of the built-in support for ffmpeg filters https://ffmpeg.org/ffmpeg-filters.html#Audio-Filters

   For example, quite often, it is necessary to normalize audio data. To do this, you can use the dynaudnorm filter. http://underpop.online.fr/f/ffmpeg/help/dynaudnorm.htm.gz by enabling additional processing and adding a filter to the list as shown below:

    func setupAudioCapture() {
        ...
        audioCaptureConfig.advancedAudioInputEnablePreprocessing = 1;
        audioCaptureConfig.advancedAudioInputPreprocessingFilters = [[
        //"dynaudnorm" : "framelen=23:gausssize=5:peak=0.80:maxgain=10:targetrms=1"
        "dynaudnorm" : "framelen=160:gausssize=5"
        ]]
        ...
    }

Recommendations

1. To achieve the minimum delay of the backward audio stream and the stability of the camera, it is recommended to start the work of the backward audio channel immediately at the start of stream playback and stop on closing. And data transfer is regulated by the method - muteMicrophone

Documentation and examples

1. For more information about the backward audio channel, see the wiki https://github.com/mts-ai/VSAAS.Media.SDK.iOS/wiki

2. A example of working with the backward audio channel can be found here https://github.com/mts-ai/VSAAS.Media.SDK.iOS/tree/master/CloudSDK/src/streamland-player