TaskCancelledException in RemoteOrchestrationServiceClient

[vandanavisw]

We are using SF backend and getting TaskCancelledException while trying to resolve the partitions in RemoteOrchestrationClient in 1 of the many Prod regions. We have an http timeout of 10sec while creating the TaskhubClient and those time out too. This fails to start any DTF operation in that region and fails the Task.
This is the stack trace:
{"ClassName":"System.Threading.Tasks.TaskCanceledException","Message":"The request was canceled due to the configured HttpClient.Timeout of 10 seconds elapsing.","Data":null,"InnerException":{"ClassName":"System.TimeoutException","Message":"A task was canceled.","Data":null,"InnerException":{"ClassName":"System.Threading.Tasks.TaskCanceledException","Message":"A task was canceled.","Data":null,"InnerException":null,"HelpURL":null,"StackTraceString":" at System.Threading.Tasks.TaskCompletionSourceWithCancellation1.WaitWithCancellationAsync(CancellationToken cancellationToken)\r\n at System.Net.Http.HttpConnectionPool.GetHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)\r\n at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)\r\n at System.Net.Http.DiagnosticsHandler.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)\r\n at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)\r\n at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)","RemoteStackTraceString":null,"RemoteStackIndex":0,"ExceptionMethod":null,"HResult":-2146233029,"Source":"System.Private.CoreLib","WatsonBuckets":null},"HelpURL":null,"StackTraceString":null,"RemoteStackTraceString":null,"RemoteStackIndex":0,"ExceptionMethod":null,"HResult":-2146233083,"Source":null,"WatsonBuckets":null},"HelpURL":null,"StackTraceString":" at DurableTask.AzureServiceFabric.Remote.RemoteOrchestrationServiceClient.ExecuteRequestWithRetriesAsync(String instanceId, Func2 requestAsync, CancellationToken cancellationToken) in C:\\__w\\1\\s\\src\\Services\\DurableTask.AzureServiceFabric\\Remote\\RemoteOrchestrationServiceClient.cs:line 403\r\n at DurableTask.AzureServiceFabric.Remote.RemoteOrchestrationServiceClient.PutJsonAsync(String instanceId, String fragment, Object object, CancellationToken cancellationToken) in C:\\__w\\1\\s\\src\\Services\\DurableTask.AzureServiceFabric\\Remote\\RemoteOrchestrationServiceClient.cs:line 317\r\n at DurableTask.AzureServiceFabric.Remote.RemoteOrchestrationServiceClient.CreateTaskOrchestrationAsync(TaskMessage creationMessage, OrchestrationStatus[] dedupeStatuses) in C:\\__w\\1\\s\\src\\Services\\DurableTask.AzureServiceFabric\\Remote\\RemoteOrchestrationServiceClient.cs:line 101\r\n at DurableTask.Core.TaskHubClient.InternalCreateOrchestrationInstanceWithRaisedEventAsync(String orchestrationName, String orchestrationVersion, String orchestrationInstanceId, Object orchestrationInput, IDictionary2 orchestrationTags, OrchestrationStatus[] dedupeStatuses, String eventName, Object eventData, Nullable1 startAt) in /_/src/DurableTask.Core/TaskHubClient.cs:line 622

What is a possible solution to this issue? Do we need additional retries while trying to start the Orchestration, so that it retries the request on a different partition?

[cgillum]

@shankarsama