Disconnect with AWS IoT Gracefully

[law-ko]

Hello,

We are trying to disconnect with AWS IoT MQTT broker after the provisioning session is complete. However, it seems like it will crash due to MQTT_ProcessLoop is still running.

We are using xTlsDisconnect() to disconnect.

What is the good way to disconnect correctly and reconnect to AWS IoT again?

[Skptak]

Hey @law-ko, thanks for reaching out about this issue as well. Is this by any chance related to your other issue #44?
But to follow the issue you are facing, you're saying that you're connecting to the IoT MQTT broker, and then disconnecting, and then seeing a crash? Do you by any chance have a stack trace showing what the flow of execution for this looks like?

[Skptak]

Hey, if this is still an issue you are facing you may want to consider trying the menuconfig option I mentioned in #46 here

[law-ko]

Hi @Skptak ,

We further investigated into this and still believes the MQTTAgent_CommandLoop might be running in the background which leads to a LoadProhibited crash when calling xTlsDisconnect(pxNetworkContext).

Here's our code of trying to disconnect from AWS IoT:

/* If a connection was previously established, close it to free memory. */
if( ( pxNetworkContext != NULL ) && ( pxNetworkContext->pxTls != NULL ) )
{
    ESP_LOGW( TAG, "Attempting to close previous TLS connection." );
    xTlsDisconnect( pxNetworkContext ); // seems like crash happens here due to pxNetworkContext
    vTaskDelay(500 / portTICK_PERIOD_MS);
    ESP_LOGI( TAG, "TLS connection was disconnected." );
    vTaskDelay(500 / portTICK_PERIOD_MS);
}

ESP_LOGW(TAG, "Before BackoffAlgorithm_InitializeParams");
BackoffAlgorithm_InitializeParams( &xReconnectParams,
                                   configRETRY_BACKOFF_BASE_MS,
                                   configRETRY_MAX_BACKOFF_DELAY_MS,
                                   BACKOFF_ALGORITHM_RETRY_FOREVER );
ESP_LOGW(TAG, "After BackoffAlgorithm_InitializeParams");

do
{
    ESP_LOGW( TAG, "Attempting to connect to MQTT broker via xTlsConnect" );
    xTlsRet = xTlsConnect( pxNetworkContext );
    vTaskDelay(500 / portTICK_PERIOD_MS);
    ESP_LOGW( TAG, "After xTlsConnect" );

    if( xTlsRet == TLS_TRANSPORT_SUCCESS )
    {
        eMqttRet = prvCoreMqttAgentConnect( xCleanSession );

        if( eMqttRet != MQTTSuccess )
        {
            ESP_LOGE( TAG,
                      "MQTT_Status: %s",
                      MQTT_Status_strerror( eMqttRet ) );
        }
    }

    if( eMqttRet != MQTTSuccess )
    {
        xTlsDisconnect( pxNetworkContext );
        xBackoffRet = prvBackoffForRetry( &xReconnectParams );
    }
} while( ( eMqttRet != MQTTSuccess ) && ( xBackoffRet == pdPASS ) );

Below is the crash log:

W (64145) core_mqtt_agent_manager: Attempting to close previous TLS connection.

Guru Meditation Error: Core  1 panic'ed (LoadProhibited). Exception was unhandled.

Core  1 register dump:
PC      : 0x402082a6  PS      : 0x00060230  A0      : 0x8010e6ec  A1      : 0x3f814a60  
0x402082a6: esp_tls_conn_write at /Users/esp/esp-idf/components/esp-tls/esp_tls.c:135

A2      : 0x00000000  A3      : 0x3f814adc  A4      : 0x00000002  A5      : 0x00000001  
A6      : 0xb33fffff  A7      : 0xb33fffff  A8      : 0x00000700  A9      : 0x3f814a20  
A10     : 0x3ffc3ad4  A11     : 0x00060420  A12     : 0x00000004  A13     : 0x00060423  
A14     : 0xb33fffff  A15     : 0xb33fffff  SAR     : 0x00000008  EXCCAUSE: 0x0000001c  
EXCVADDR: 0x00000728  LBEG    : 0x4000c46c  LEND    : 0x4000c477  LCOUNT  : 0x00000000  
0x4000c46c: memset in ROM

0x4000c477: memset in ROM



Backtrace: 0x402082a3:0x3f814a60 0x4010e6e9:0x3f814a80 0x40110259:0x3f814aa0 0x401106d5:0x3f814ad0 0x401111b9:0x3f814b00 0x4011164e:0x3f814b80 0x40113096:0x3f814ba0 0x400df239:0x3f814c00 0x400949c2:0x3f814c20
0x402082a3: esp_tls_conn_write at /Users/esp/esp-idf/components/esp-tls/esp_tls.c:135

0x4010e6e9: espTlsTransportSend at /Users/esp-git/esp-matter/libraries/esp-aws-iot/libraries/coreHTTP/port/network_transport/network_transport.c:79

0x40110259: sendBuffer at /Users/esp-git/esp-matter/libraries/esp-aws-iot/libraries/coreMQTT/coreMQTT/source/core_mqtt.c:861

0x401106d5: MQTT_Ping at /Users/esp-git/esp-matter/libraries/esp-aws-iot/libraries/coreMQTT/coreMQTT/source/core_mqtt.c:2900

0x401111b9: handleKeepAlive at /Users/esp-git/esp-matter/libraries/esp-aws-iot/libraries/coreMQTT/coreMQTT/source/core_mqtt.c:1352
 (inlined by) receiveSingleIteration at /Users/esp-git/esp-matter/libraries/esp-aws-iot/libraries/coreMQTT/coreMQTT/source/core_mqtt.c:1698

0x4011164e: MQTT_ProcessLoop at /Users/esp-git/esp-matter/libraries/esp-aws-iot/libraries/coreMQTT/coreMQTT/source/core_mqtt.c:3065

0x40113096: processCommand at /Users/esp-git/esp-matter/libraries/esp-aws-iot/libraries/coreMQTT-Agent/coreMQTT-Agent/source/core_mqtt_agent.c:611
 (inlined by) MQTTAgent_CommandLoop at /Users/esp-git/esp-matter/libraries/esp-aws-iot/libraries/coreMQTT-Agent/coreMQTT-Agent/source/core_mqtt_agent.c:1059

0x400df239: prvMQTTAgentTask at /Users/esp-git/esp-matter/examples/outlet/main/aws_iot/networking/mqtt/core_mqtt_agent_manager.c:530

0x400949c2: vPortTaskWrapper at /Users/esp/esp-idf/components/freertos/FreeRTOS-Kernel/portable/xtensa/port.c:162

And for esp_tls.c at line 135:

ssize_t esp_tls_conn_write(esp_tls_t *tls, const void  *data, size_t datalen)
{
    return tls->write(tls, (char *)data, datalen); // line 135

}

How should we call xTlsDisconnect gracefully? We do not want to kill MQTTAgent as we are reconnecting afterwards.

[Skptak]

Hey @law-ko, thanks for letting me know that you're still seeing this issue!
There are a couple different issues being opened in regards to the mqtt process loop at this point

FreeRTOS/coreMQTT-Agent#104, #63, and #44

The team is taking a look into all of these. Unfortunately, it can be hard to pinpoint exactly what is going wrong since many of these issues can be hard to reproduce. I'm going to be trying to replicate some of these issues over the next few days and will see if I can get back to you about it

Thanks in advance for your patience!

[aggarg]

I think your assessment is correct that you are disconnecting the underlying TLS connection without stopping the MQTT Agent.

Assuming that your implementation is similar to core_mqtt_agent_manager.c which uses xNetworkEventGroup to control when the MQTT agent task should call MQTTAgent_CommandLoop, you can stop the MQTT Agent like the following before disconnecting the TLS connection:

        /* If a connection was previously established, close it to free memory. */
        if( ( pxNetworkContext != NULL ) && ( pxNetworkContext->pxTls != NULL ) )
        {
            /* Ensure that the MQTT Agent task stops calling MQTTAgent_CommandLoop. */
            xEventGroupClearBits( xNetworkEventGroup,
                                  CORE_MQTT_AGENT_CONNECTED_BIT );
            xEventGroupSetBits( xNetworkEventGroup,
                                CORE_MQTT_AGENT_DISCONNECTED_BIT );

            /* Close the underlying TLS connection. */
            ESP_LOGW( TAG, "Attempting to close previous TLS connection." );
            xTlsDisconnect( pxNetworkContext );

            ESP_LOGI( TAG, "TLS connection was disconnected." );
        }

If your implementation is different, please share your complete implementation.

[AniruddhaKanhere]

Hello,

I am closing this issue as there hasn't been any update.
Please feel free to reopen this issue or create a new one.

Thanks