Issue integrating new decoder

[dgrisham]

Hi!

I'm working on integrating the FDK AAC decoder into Miniaudio. I successfully got it working as a custom decoder backend as per the libvorbis example. Now I'm generating bindings to use the Miniaudio library from a different programming language. To this end, I integrated my updates with the 'split' version of miniaudio. I also attempted to add the AAC/M4A support directly into the library rather than as a custom backend, so it would all be bundled together / 'just work' from the perspective of my client code.

My attempt is here, where the MA_HAS_M4A macro surrounds the relevant code additions. The client code I'm using to run this is the same as the README in this Jai example. That code works with the provided .mp3 file, but not with my .m4a file that I had working in my original Miniaudio implementation with just C and integrating my decoder as a custom backend.

The issue I'm having is at the ma_engine_play_sound call -> in the ma_data_converter_get_heap_layout function, which returns MA_INVALID_ARGS at this check:

    if (pConfig->channelsIn == 0 || pConfig->channelsOut == 0) { // * both conditions are true in my example
        return MA_INVALID_ARGS;
    }

My first assumption was that the code that integrates custom decoder backends handles any (non-essential) unimplemented functions for you, and so that may not be happening here, which is why my code worked as a custom decoder but not when integrated directly. However, from what I can tell the unimplemented functions are handled for 'supported' decoders as well. For reference, here's my decoding backend vtable definition:

static ma_decoding_backend_vtable g_ma_decoding_backend_vtable_fdkaac =
{
    ma_decoding_backend_init__fdkaac,
    ma_decoding_backend_init_file__fdkaac,
    NULL, /* onInitFileW() */
    NULL, /* onInitMemory() */
    ma_decoding_backend_uninit__fdkaac
};

So my questions are:

• Is there something else I'm missing here, that would allow me to integrate my decoder directly in this way?
• Or: is there a way to integrate a 'custom backend', but in a way that is completely inside the miniaudio library I'm building and without touching the client code in Jai?

[dgrisham]

Got a bit more info here:

My decoder seems to work fine if I open with ma_decoder_init_file(), but not with ma_resource_manager_data_source_init(). Not yet certain what code path is relevant to the latter that doesn't initialize my decoder properly.

[mackron]

Sorry for the delay. What are you returning from your custom decoder's get_data_format() callback?

[dgrisham]

Function def is below - the gist is that it defaults to empty values, then fills with values from the pAAC->info struct if it can. And pAAC->info will only be defined if either the read_pcm_frames or init_file functions are called, because we have to decode at least one frame to get that information in this case.

MA_API ma_result ma_fdkaac_get_data_format(ma_fdkaac* pAAC, ma_format* pFormat, ma_uint32* pChannels, ma_uint32* pSampleRate, ma_channel* pChannelMap, size_t channelMapCap)
{
    /* Defaults for safety. */
    if (pFormat != NULL) {
        *pFormat = ma_format_unknown;
    }
    if (pChannels != NULL) {
        *pChannels = 0;
    }
    if (pSampleRate != NULL) {
        *pSampleRate = 0;
    }
    if (pChannelMap != NULL) {
        MA_ZERO_MEMORY(pChannelMap, sizeof(*pChannelMap) * channelMapCap);
    }

    if (pAAC == NULL) {
        return MA_INVALID_OPERATION;
    }

    if (pFormat != NULL) {
        *pFormat = pAAC->format;
    }

    #if !defined(MA_NO_FDKACC)
    {
        if (pChannels != NULL) {
            if (pAAC->info) *pChannels = pAAC->info->numChannels;
        }

        if (pSampleRate != NULL) {
            if (pAAC->info) *pSampleRate = pAAC->info->sampleRate;
        }

        if (pChannelMap != NULL) {
            ma_channel_map_init_standard(ma_standard_channel_map_fdkaac, pChannelMap, channelMapCap, *pChannels);
        }

        return MA_SUCCESS;
    }
    #else
    {
        /* fdkaac is disabled. Should never hit this since initialization would have failed. */
        MA_ASSERT(MA_FALSE);
        return MA_NOT_IMPLEMENTED;
    }
    #endif
}

[mackron]

Are you able to post a full call stack at the time of the error? You were saying both pConfig->channelsIn and pConfig->channelsOut are 0? This sounds very suspicious. Are you able to post the contents of the entire config so I can see if anything is valid in that?

I looked at your code (I somehow missed your mention of it earlier) and I can't see any glaringly obvious errors. There is one error, however (not sure if it's causing this particular issue though), and that is your custom channel map ma_standard_channel_map_fdkaac. For that to work you would need to update ma_channel_map_init_standard_channel() to account for your channel map. I would instead recommend you just output the channel map directly from ma_fdkaac_get_data_format() and skip ma_channel_map_init_standard() altogether. However, having said all that, I would highly recommend you do not integrate your decoder like you're doing. First of all, that split version of miniaudio is auto-generated from a tool and the next time you update it will just get overwritten. The other reason is that I did a big refactor of that part of the code in the dev-0.12 branch and you'll be in a lot of pain when you update to that. Keeping your decoder in a separate file will make it much easier to update.

[dgrisham]

You were right, I've worked on implementing the onInit function with callbacks, there's just one sticking point with the seek callback. The ffmpeg library requires that the seek callback returns the filesize without seeking whenever its passes AVSEEK_SIZE as the whence parameter. And it seems that the avformat_open_input() function, which does the main intialization/setup within onInit, uses AVSEEK_SIZE when finding the moov atom within an M4A file. There isn't a direct way to get the filesize from the avformat library, and I'm unable to use an fseek()/ftell() type of approach on the buffer loaded by avformat.

So I think I need to get the filesize from Miniaudio if possible. My program only loads files (never an internet radio stream/etc.). Is there a straightforward way to get the filesize from within onInit?

[mackron]

Sorry for the delay. Not quite sure I follow. miniaudio doesn't care about the implementation details of the backend, and it doesn't care about the size of the file. So the details of AVSEEK_SIZE should be totally irrelevant from the perspective of miniaudio. Why are you needing the size of the file from within onInit?

[dgrisham]

So, I've implemented the seek callback as follows:

static int64_t ma_fdkaac_of_callback__seek(void* pUserData, int64_t offset, int whence)
{
    ma_fdkaac* pAAC = (ma_fdkaac*)pUserData;
    ma_result result;
    ma_seek_origin origin;

    if (whence == SEEK_SET) {
        origin = ma_seek_origin_start;
    } else if (whence == SEEK_END) {
        origin = ma_seek_origin_end;
    } else if (whence == AVSEEK_SIZE) {
        // need to return the file size here, and not actually seek
        return -1;
    } else {
        origin = ma_seek_origin_current;
    }

    result = pAAC->onSeek(pAAC->pReadSeekTellUserData, offset, origin);
    if (result != MA_SUCCESS) {
        if (result == MA_AT_END) {
            return pAAC->in->pb->pos;
        }
        return -1;
    }

    return pAAC->in->pb->pos;
}

I'm basing that off of the included libvorbis example, with the AVSEEK_SIZE addition coming from https://www.codeproject.com/Tips/489450/Creating-Custom-FFmpeg-IO-Context, where it says:

The whence-parameter has one more option than fseek: AVSEEK_SIZE. When this option is passed to the seek function
it should return the file size (if possible). If its not possible, the function may return and do nothing -1.

Even though it says AVSEEK_SIZE is optional, it seems that I need to implement it so that the library can properly open the stream. When I simply return -1, I get the following error when opening an m4a/aac file:

[mov,mp4,m4a,3gp,3g2,mj2 @ 0x36ab57c0] moov atom not found
avformat_open_input failed: Invalid data found when processing input

So to test that it would work if I returned the actual file size, I picked a test file and hardcoded the function to return the exact number of bytes in that file, and everything initialized properly. My loose understanding is that it uses the seek callback to get the filesize when attempting to parse the moov atom that it says it's not finding.

Hopefully that makes sense -- here's the onInit function that calls avformat_open_input and prints the error shown above:

MA_API ma_result ma_fdkaac_init(ma_read_proc onRead, ma_seek_proc onSeek, ma_tell_proc onTell, void* pReadSeekTellUserData, const ma_decoding_backend_config* pConfig, const ma_allocation_callbacks* pAllocationCallbacks, ma_fdkaac* pAAC)
{
    ma_result result;

    (void)pAllocationCallbacks;

    result = ma_fdkaac_init_internal(pConfig, pAAC);
    if (result != MA_SUCCESS) {
        return result;
    }

    if (onRead == NULL || onSeek == NULL) {
        return MA_INVALID_ARGS; /* onRead and onSeek are mandatory. */
    }

    pAAC->onRead = onRead;
    pAAC->onSeek = onSeek;
    pAAC->onTell = onTell;
    pAAC->pReadSeekTellUserData = pReadSeekTellUserData;

    #if !defined(MA_NO_FDKACC)
    {
        const int iBufSize = 32 * 1024;
        char* pBuffer = (char*)av_malloc(iBufSize);
        AVIOContext* pIOCtx = avio_alloc_context(pBuffer, iBufSize, 0, pAAC, ma_fdkaac_of_callback__read, 0, ma_fdkaac_of_callback__seek);
        if (pIOCtx == NULL) {
            fprintf(stderr, "Failed to allocated AVIO context\n");
            return MA_ERROR;
        }

        pAAC->in = avformat_alloc_context();
        pAAC->in->pb = pIOCtx;

        // Determining the input format:
        int ulReadBytes = ma_fdkaac_of_callback__read((void*)pAAC, pBuffer, iBufSize);
        if (ulReadBytes < 0) {
            fprintf(stderr, "Initial read failed\n");
            return MA_ERROR;
        }

        // Don't forget to reset the data pointer back to the beginning!
        if (ma_fdkaac_of_callback__seek((void*)pAAC, 0, SEEK_SET) < 0) {
            fprintf(stderr, "Initial seek failed\n");
            return MA_ERROR;
        }

        // Now we set the ProbeData-structure for av_probe_input_format:
        AVProbeData probeData;
        probeData.buf = pBuffer;
        probeData.buf_size = ulReadBytes;
        probeData.filename = "";

        // Determine the input format:
        pAAC->in->iformat = av_probe_input_format(&probeData, 1);
        pAAC->in->flags = AVFMT_FLAG_CUSTOM_IO;

	int ret;
        ret = avformat_open_input(&pAAC->in, NULL, NULL, NULL);
        if (ret != 0) {
            char buf[100];
            av_strerror(ret, buf, sizeof(buf));
            fprintf(stderr, "avformat_open_input failed: %s\n", buf);
            return MA_INVALID_FILE;
        }

    	AVStream *st = NULL;
    	UINT input_length;
    	AAC_DECODER_ERROR err;

        for (i = 0; i < pAAC->in->nb_streams && !st; i++) {
            if (pAAC->in->streams[i]->codecpar->codec_id == AV_CODEC_ID_AAC)
                st = pAAC->in->streams[i];
        }
        if (!st) {
            fprintf(stderr, "No AAC stream found\n");
            return MA_INVALID_DATA;
        }
        if (!st->codecpar->extradata_size) {
            fprintf(stderr, "No AAC ASC found\n");
            return MA_INVALID_DATA;
        }

        if (pAAC->handle == NULL) {
     		pAAC->handle = aacDecoder_Open(TT_MP4_RAW, 1);
        }
        pAAC->pcm_frame_cursor = 0;
        pAAC->st = st;
    	input_length = st->codecpar->extradata_size;
        err = aacDecoder_ConfigRaw(pAAC->handle, &st->codecpar->extradata, &input_length);
        if (err != AAC_DEC_OK) {
            fprintf(stderr, "Unable to decode the ASC\n");
            return MA_INVALID_DATA;
        }
        pAAC->info = NULL;

    	pAAC->decode_buf_size = 8*2048*sizeof(INT_PCM); // larger than we probably need (maybe 2048 * 2 * sizeof(INT_PCM) is more realistic?)
    													// HE-AAC maxes at 2048 PCM frames in one AAC frame * 2 channels (maybe the example
    													// used 8 for 7.1 surround?)
        pAAC->decode_buf = (INT_PCM*)ma_malloc(pAAC->decode_buf_size, pAllocationCallbacks);
        pAAC->decode_buf_start = -1; // -1 means we don't currently have any valid data in the buffer

        // loads one frame into the buffer and initializes pAAC->info (so we have number of channels / etc.)
        return decode_one_aac_frame(pAAC);
    }
    #else
    {
        /* fdkaac is disabled. */
        return MA_NOT_IMPLEMENTED;
    }
    #endif
}

[mackron]

OK, I see what you're saying. I thought you were talking about the implementation of miniaudio's onSeek callback, not the FFmpeg seek callback. I don't think there will be an elegant way to solve this. It's messy, but one thing you could do is calculate the size of the file yourself in terms of onRead (that's miniaudio's ma_read_proc, not FFmpeg's callback). Run that in a loop, with the output buffer just being a temp buffer on the stack (you'll ignore that data) and then accumulate the number of bytes read from each call to onRead. Then terminate the loop when onRead returns MA_AT_END, or the output byte count is less than what you requested. Then seek back to the start with onSeek. Then you should have a valid value to return value for AVSEEK_SIZE. You would invoke all of this before initializing your FFmpeg object. I think the seek/tell pattern should also work, so long as you do it with the miniaudio callbacks, and before initializing the FFmpeg object.

[dgrisham]

That worked, thank you! I used the seek/tell approach in onInit before initializing the ffmpeg context/decoder, then stored the file size in my decoder object + return it whenever AVSEEK_SIZE is passed. Here's the code excerpt:

MA_API ma_result ma_fdkaac_init(ma_read_proc onRead, ma_seek_proc onSeek, ma_tell_proc onTell, void* pReadSeekTellUserData, const ma_decoding_backend_config* pConfig, const ma_allocation_callbacks* pAllocationCallbacks, ma_fdkaac* pAAC)
{
    // ...
    
    #if !defined(MA_NO_FDKACC)
    {
        result = onSeek(pReadSeekTellUserData, 0, SEEK_END);
        if (result != MA_SUCCESS) {
            fprintf(stderr, "Failed to seek to end of file with onSeek\n");
            return MA_ERROR;
        }
        ma_int64 fsize;
        result = onTell(pReadSeekTellUserData, &fsize);
        printf("fsize: %d\n", fsize);
        if (result != MA_SUCCESS) {
            fprintf(stderr, "Failed to get filesize with onTell\n");
            return MA_ERROR;
        }
        pAAC->fsize = fsize;
        result = onSeek(pReadSeekTellUserData, 0, SEEK_SET);
        if (result != MA_SUCCESS) {
            fprintf(stderr, "Failed to seek back to start of file with onSeek\n");
            return MA_ERROR;
        }
        // ...
    }
    // ...
}