Operation error not handled correctly by a long-running operation

[mikhailshilkov]

I'm using go-autorest to provision a long running resource under AzureActiveDirectory. It fails but the error message is swallowed and a faulty error message is returned.

Here is an example.

The initial PUT operation returns a 202 with the link to the operation:

PUT /subscriptions/<sub>/resourceGroups/my-rg1879968a/providers/Microsoft.AzureActiveDirectory/b2cDirectories/newB2Ctenant3.onmicrosoft.com?api-version=2019-01-01-preview
{"location":"europe","properties":{"createTenantProperties":{"countryCode":"US","displayName":"Contoso"}},"sku":{"name":"Standard","tier":"A0"}}

    HTTP/1.1 202 Accepted
    Date: Thu, 27 May 2021 15:27:09 GMT
    Location: https://management.azure.com/subscriptions/<sub>/providers/microsoft.azureactivedirectory/operations/59cbac1d-f6a7-4172-ac72-7d6d15ad1591?api-version=2019-01-01-preview
    Retry-After: 60
    Strict-Transport-Security: max-age=31536000; includeSubDomains
    X-Content-Type-Options: nosniff
    X-Ms-Correlation-Request-Id: 2cb2a03c-e971-42ec-b3b7-a86798b4f124
    X-Ms-Gateway-Requestid: 597a2e87-c701-46ec-bb87-2ab49149eede
    X-Ms-Ratelimit-Remaining-Subscription-Resource-Requests: 99
    X-Ms-Request-Id: 1b5e99f6-7431-41c6-b49f-329f81c06892
    X-Ms-Routing-Request-Id: WESTEUROPE:20210527T152710Z:2cb2a03c-e971-42ec-b3b7-a86798b4f124
    X-Ms-Version: 1.0.2097.0
    X-Ua-Compatible: IE=edge
    X-Xss-Protection: 1; mode=block
    Content-Length: 0

The operation returns a 200 with an error message:

GET https://management.azure.com/subscriptions/<sub>/providers/microsoft.azureactivedirectory/operations/27cbcce7-4918-46f4-a362-4759c13dd67c?api-version=2019-01-01-preview

HTTP/1.1 200 OK
    Content-Type: application/json; charset=utf-8
    Date: Thu, 27 May 2021 15:29:11 GMT
    X-Content-Type-Options: nosniff
    X-Ms-Correlation-Request-Id: d1462bf9-d4de-4313-ae23-d9da0def59b6
    X-Ms-Gateway-Requestid: c2c1cffc-036f-4904-a0a9-f57c8741685c
    X-Ms-Ratelimit-Remaining-Subscription-Resource-Requests: 297
    X-Ms-Request-Id: 94d5f561-e18d-4764-9ce3-848bbc739d0e
    X-Ms-Routing-Request-Id: WESTEUROPE:20210527T152911Z:d1462bf9-d4de-4313-ae23-d9da0def59b6
    X-Ms-Version: 1.0.2097.0
    X-Ua-Compatible: IE=edge
    X-Xss-Protection: 1; mode=block
    264
    {"subscriptionId":"<sub>","id":"/<sub>/providers/microsoft.azureactivedirectory/operations/27cbcce7-4918-46f4-a362-4759c13dd67c","name":"27cbcce7-4918-46f4-a362-4759c13dd67c","status":"Failed","startTime":"05/27/2021 15:28:11","endTime":"05/27/2021 15:28:11","error":{"code":"BadRequest","message":"Cannot create resource with location: 'europe'. Please use location: 'United States' to create resource for b2c directory: ''.","extendedDetails":{"affectsQos":false,"statusCode":400}},"properties":{"billingConfig":null,"tenantId":null}}

However, go-autorest then proceeds to query the resource which doesn't exist:

GET https://management.azure.com/subscriptions/<sub>/resourceGroups/my-rg1879968a/providers/Microsoft.AzureActiveDirectory/b2cDirectories/newB2Ctenant3.onmicrosoft.com?api-version=2019-01-01-preview ===================================================
    HTTP/1.1 404 Not Found
    Content-Type: application/json; charset=utf-8
    Date: Thu, 27 May 2021 15:29:11 GMT
    X-Content-Type-Options: nosniff
    X-Ms-Correlation-Request-Id: 53df5f97-c044-41ad-bfb7-b51bb43b80e8
    X-Ms-Gateway-Requestid: 9b238fc6-ae66-492a-b880-23e1a54d7278
    X-Ms-Ratelimit-Remaining-Subscription-Resource-Requests: 298
    X-Ms-Request-Id: db009ca2-66d7-4951-82ef-69a5b12832e9
    X-Ms-Routing-Request-Id: WESTEUROPE:20210527T152911Z:53df5f97-c044-41ad-bfb7-b51bb43b80e8
    X-Ms-Version: 1.0.2097.0
    X-Ua-Compatible: IE=edge
    X-Xss-Protection: 1; mode=block
    {"error":{"code":"NotFound","message":"Resource with ID '0<sub>/my-rg1879968a/newB2Ctenant3.onmicrosoft.com' does not exist.","target":"resource"}}

This last error is returned as a Go error, which confuses our users.

[joedevgee]

It's been 6+ months. Can somebody change the 2xx response to a 4xx? this mysterious error is really confusing to end users

[mikhailshilkov]

Another example of the same issue with storage accounts: pulumi/pulumi-azure-native#1138 (comment)

@jhendrixMSFT sorry for tagging directly, but could you take a look?

[jhendrixMSFT]

Sorry for the delay.

Do you have an example of how you're using go-autorest in this context? IIRC, you are using it outside of our SDK.

Also, we will be deprecating go-autorest in the coming year (exact date is still being worked out). It will be replaced by azcore. If you're interested in the LRO-specific bits you can find them here.

[mikhailshilkov]

We basically do this:

resp, err = autorest.SendWithSender(
  k.client,
  prepReq,
  azure.DoRetryWithRegistration(k.client),
)
if err != nil {
  return nil, false, err
}

future, err := azure.NewFutureFromResponse(resp)
if err != nil {
	return nil, created, err
}
err = future.WaitForCompletionRef(ctx, k.client)
if err != nil {
	return nil, created, err
}
resp, err = future.GetResult(k.client)
if err != nil {
	return nil, created, err
}

Is that what you were asking?

[jhendrixMSFT]

Thanks this is helpful.

I believe that the service is in error here, not following the Azure RPC spec. I will follow up on this, but it means that there's no good workaround at present other than writing a custom future/poller to handle this case. :(

[mikhailshilkov]

What's the exact violation here? Would it be a bad idea to relax the library and handle these cases? They seem to be coming from multiple Azure services - see linked issues.

writing a custom future/poller to handle this case

Do you have an example of this anywhere?

Would azcore be any different in this regards?

[jhendrixMSFT]

In this case, based on the terminal response, I believe that the initial response to the PUT should have returned the polling URL in the Azure-AsyncOperation header, not Location header (see here for how I came to that conclusion). The services must follow the Azure RPC spec as all of our SDKs depend on this. I'm still waiting for confirmation that the service is indeed malformed.

[jhendrixMSFT]

Issue has been reported to the service team. I'll post when I have more info.

[pierskarsenbarg]

@jhendrixMSFT any update on this?

[jhendrixMSFT]

According to the service team, this was fixed in API version 2021-04-01.

[mikhailshilkov]

I've just tried with 2021-04-01 and got exactly the same issue, unfortunately