Failed to assume role with OIDC: AggregateError [ETIMEDOUT]

[zafodB]

Describe the bug

Occasionally, when OIDC authentication is used, the tasks fail to authenticate (presumably due to a timeout) and fail the pipeline. This is usually a transient issue that is fixed by a retry. However, this creates unnecessary reruns, which is particularly cumbersome in long CI/CD pipelines.

To reproduce

1. Configure this extension to use OIDC authentication by assuming a role from AWS
2. Configure a task to perform an authenticated action on AWS. In our use-case it is an AWS CloudFormation Create/Update Stack task, but we suspect the issue may affect other tasks as well
3. Use the task in a pipeline. In 4 out of 5 cases the task successfully assumes the role via OIDC and performs the action (CloudFormation stack update) as expected. In 1 out of 5 cases (approximately) the task fails to assume the role via OIDC and fails. Further reruns without any configuration changes then usually succeed again.
4. It has not been established under what conditions the task fails. It happens randomly across short and long CI/CD pipelines, with different settings for credentials validity and other configuration differences. It seems to appear at random, possibly linked to some lower-level (network?) timeouts.

Expected behavior

In step 3. we expect the task to succeed 5 out of 5 times, if no configuration changes are made.

Your Environment

• On-prem or cloud based?: Cloud
• Azure DevOps version: Version Dev19.M250 (AzureDevOps_M250_20250205.2)
• AWS Toolkit for Azure DevOps version: 1.19.0 (Latest)

Additional context

Azure Pipeline Logs when failure (~1 out of 5 runs under the same configuration)

2025-02-06T22:10:51.5442197Z ##[section]Starting: SAMPLE_TASK
2025-02-06T22:10:51.5451502Z ==============================================================================
2025-02-06T22:10:51.5451670Z Task         : AWS CloudFormation Create/Update Stack
2025-02-06T22:10:51.5451770Z Description  : Creates a new AWS CloudFormation stack or updates the stack if it exists.
2025-02-06T22:10:51.5451908Z Version      : 1.19.0
2025-02-06T22:10:51.5452003Z Author       : Amazon Web Services
2025-02-06T22:10:51.5452114Z Help         : Please refer to [AWS CloudFormation User Guide](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/) for more details on working with AWS CloudFormation.

More information on this task can be found in the [task reference](https://docs.aws.amazon.com/vsts/latest/userguide/cloudformation-create-update.html).

####Task Permissions
This task requires permissions to call the following AWS service APIs (depending on selected task options, not all APIs may be used):
* cloudformation:CreateChangeSet
* cloudformation:CreateStack
* cloudformation:DeleteChangeSet
* cloudformation:DescribeChangeSet
* cloudformation:DescribeStacks
* cloudformation:DescribeStackResources
* cloudformation:ExecuteChangeSet
* cloudformation:UpdateStack

The task may also require permissions to upload your application template to the specified Amazon S3 bucket.
2025-02-06T22:10:51.5452874Z ==============================================================================
2025-02-06T22:10:52.1452358Z Configuring credentials for task
2025-02-06T22:10:52.1462225Z ...configuring AWS credentials from service endpoint 'REDACTED-BY-ZAFODB'
2025-02-06T22:10:52.1463156Z Skipping Instance profile, we have OIDC enabled
2025-02-06T22:10:52.1473677Z ...configuring AWS credentials from service endpoint 'REDACTED-BY-ZAFODB'
2025-02-06T22:10:52.1474191Z Getting OIDC Token...
2025-02-06T22:10:52.4597307Z Failed to assume role with OIDC: AggregateError [ETIMEDOUT]:
2025-02-06T22:10:52.4598475Z     at internalConnectMultiple (node:net:1118:18)
2025-02-06T22:10:52.4599162Z     at internalConnectMultiple (node:net:1186:5)
2025-02-06T22:10:52.4599542Z     at Timeout.internalConnectMultipleTimeout (node:net:1712:5)
2025-02-06T22:10:52.4599816Z     at listOnTimeout (node:internal/timers:583:11)
2025-02-06T22:10:52.4600271Z     at process.processTimers (node:internal/timers:519:7) {
2025-02-06T22:10:52.4600528Z   code: 'ETIMEDOUT',
2025-02-06T22:10:52.4600744Z   [errors]: [Array]
2025-02-06T22:10:52.4600936Z }
2025-02-06T22:10:52.4609833Z ...configuring AWS credentials from service endpoint 'REDACTED-BY-ZAFODB'
2025-02-06T22:10:52.4613907Z ...endpoint defines role-based credentials for role ***.
2025-02-06T22:10:52.4619076Z Configuring region for task
2025-02-06T22:10:52.4621156Z ...configured to use region eu-central-1, defined in task.
2025-02-06T22:10:52.4695887Z Configuring credentials for task
2025-02-06T22:10:52.4699135Z ...configuring AWS credentials from service endpoint 'REDACTED-BY-ZAFODB'
2025-02-06T22:10:52.4699662Z Skipping Instance profile, we have OIDC enabled
2025-02-06T22:10:52.4703908Z ...configuring AWS credentials from service endpoint 'REDACTED-BY-ZAFODB'
2025-02-06T22:10:52.4704205Z Getting OIDC Token...
2025-02-06T22:10:52.7789292Z OIDC Token generated: issuer: {https://vstoken.dev.azure.com/REDACTED-BY-ZAFODB} sub: {sc://REDACTED-BY-ZAFODB_service_connection}, aud: {api://REDACTED-BY-ZAFODB}
2025-02-06T22:10:52.7789693Z Configuring region for task
2025-02-06T22:10:52.7793087Z ...configured to use region eu-central-1, defined in task.
2025-02-06T22:10:52.7819001Z Assuming role via OIDC Token...
2025-02-06T22:10:53.0142132Z ...role assumed via OIDC Token: arn:aws:sts::REDACTED-BY-ZAFODB
2025-02-06T22:10:53.0144391Z Configuring region for task
2025-02-06T22:10:53.0152177Z ...configured to use region eu-central-1, defined in task.
2025-02-06T22:10:53.0273666Z Checking existence for stack REDACTED-BY-ZAFODB
2025-02-06T22:10:53.3962479Z Test for existence of stack REDACTED-BY-ZAFODB returned error: 'Error [CredentialsError]: Missing credentials in config, if using AWS_CONFIG_FILE, set AWS_SDK_LOAD_CONFIG=1
2025-02-06T22:10:53.3963165Z     at constructor.cde (/home/vsts/work/_tasks/CloudFormationCreateOrUpdateStack_7ef7cdfa-aa45-42c5-93c8-d7603643dd99/1.19.0/CloudFormationCreateOrUpdateStack.js:37:67877)
2025-02-06T22:10:53.3964000Z     at constructor.callListeners (/home/vsts/work/_tasks/CloudFormationCreateOrUpdateStack_7ef7cdfa-aa45-42c5-93c8-d7603643dd99/1.19.0/CloudFormationCreateOrUpdateStack.js:37:97340)
2025-02-06T22:10:53.3964461Z     at constructor.emit (/home/vsts/work/_tasks/CloudFormationCreateOrUpdateStack_7ef7cdfa-aa45-42c5-93c8-d7603643dd99/1.19.0/CloudFormationCreateOrUpdateStack.js:37:97048)
2025-02-06T22:10:53.3965827Z     at constructor.emitEvent (/home/vsts/work/_tasks/CloudFormationCreateOrUpdateStack_7ef7cdfa-aa45-42c5-93c8-d7603643dd99/1.19.0/CloudFormationCreateOrUpdateStack.js:38:28344)
2025-02-06T22:10:53.3966491Z     at constructor.t (/home/vsts/work/_tasks/CloudFormationCreateOrUpdateStack_7ef7cdfa-aa45-42c5-93c8-d7603643dd99/1.19.0/CloudFormationCreateOrUpdateStack.js:38:23900)
2025-02-06T22:10:53.3967116Z     at yI.runTo (/home/vsts/work/_tasks/CloudFormationCreateOrUpdateStack_7ef7cdfa-aa45-42c5-93c8-d7603643dd99/1.19.0/CloudFormationCreateOrUpdateStack.js:38:1153)
2025-02-06T22:10:53.3968472Z     at /home/vsts/work/_tasks/CloudFormationCreateOrUpdateStack_7ef7cdfa-aa45-42c5-93c8-d7603643dd99/1.19.0/CloudFormationCreateOrUpdateStack.js:38:1365
2025-02-06T22:10:53.3969065Z     at constructor.<anonymous> (/home/vsts/work/_tasks/CloudFormationCreateOrUpdateStack_7ef7cdfa-aa45-42c5-93c8-d7603643dd99/1.19.0/CloudFormationCreateOrUpdateStack.js:38:24115)
2025-02-06T22:10:53.3970082Z     at constructor.<anonymous> (/home/vsts/work/_tasks/CloudFormationCreateOrUpdateStack_7ef7cdfa-aa45-42c5-93c8-d7603643dd99/1.19.0/CloudFormationCreateOrUpdateStack.js:38:28399)
2025-02-06T22:10:53.3970545Z     at constructor.callListeners (/home/vsts/work/_tasks/CloudFormationCreateOrUpdateStack_7ef7cdfa-aa45-42c5-93c8-d7603643dd99/1.19.0/CloudFormationCreateOrUpdateStack.js:37:97445) {
2025-02-06T22:10:53.3971066Z   code: 'CredentialsError',
2025-02-06T22:10:53.3971479Z   time: 2025-02-06T22:10:53.394Z,
2025-02-06T22:10:53.3972502Z   requestId: 'REDACTED-BY-ZAFODB',
2025-02-06T22:10:53.3972821Z   statusCode: 403,
2025-02-06T22:10:53.3973094Z   retryable: false,
2025-02-06T22:10:53.3973379Z   retryDelay: 99.59529417676578,
2025-02-06T22:10:53.3973672Z   originalError: [Object]
2025-02-06T22:10:53.3973917Z }'.
2025-02-06T22:10:53.3974203Z Stack does not exist, switching to create stack mode
2025-02-06T22:10:53.3974635Z Loading template file from '/home/vsts/work/1/s/cloudformation/REDACTED-BY-ZAFODB.yml'
2025-02-06T22:10:53.3974994Z Loading template parameters from task definition
2025-02-06T22:10:53.4014134Z Successfully loaded template parameters
2025-02-06T22:10:53.4014509Z Creating stack with template file /home/vsts/work/1/s/cloudformation/REDACTED-BY-ZAFODB.yml
2025-02-06T22:10:53.4018737Z Setting capability CAPABILITY_IAM for stack
2025-02-06T22:10:53.4019077Z Setting capability CAPABILITY_NAMED_IAM for stack
2025-02-06T22:10:53.6913215Z Stack creation request failed with error: 'Missing credentials in config, if using AWS_CONFIG_FILE, set AWS_SDK_LOAD_CONFIG=1' Error [CredentialsError]: Missing credentials in config, if using AWS_CONFIG_FILE, set AWS_SDK_LOAD_CONFIG=1
2025-02-06T22:10:53.6914754Z     at constructor.cde (/home/vsts/work/_tasks/CloudFormationCreateOrUpdateStack_7ef7cdfa-aa45-42c5-93c8-d7603643dd99/1.19.0/CloudFormationCreateOrUpdateStack.js:37:67877)
2025-02-06T22:10:53.6917300Z     at constructor.callListeners (/home/vsts/work/_tasks/CloudFormationCreateOrUpdateStack_7ef7cdfa-aa45-42c5-93c8-d7603643dd99/1.19.0/CloudFormationCreateOrUpdateStack.js:37:97340)
2025-02-06T22:10:53.6918569Z     at constructor.emit (/home/vsts/work/_tasks/CloudFormationCreateOrUpdateStack_7ef7cdfa-aa45-42c5-93c8-d7603643dd99/1.19.0/CloudFormationCreateOrUpdateStack.js:37:97048)
2025-02-06T22:10:53.6919708Z     at constructor.emitEvent (/home/vsts/work/_tasks/CloudFormationCreateOrUpdateStack_7ef7cdfa-aa45-42c5-93c8-d7603643dd99/1.19.0/CloudFormationCreateOrUpdateStack.js:38:28344)
2025-02-06T22:10:53.6920776Z     at constructor.t (/home/vsts/work/_tasks/CloudFormationCreateOrUpdateStack_7ef7cdfa-aa45-42c5-93c8-d7603643dd99/1.19.0/CloudFormationCreateOrUpdateStack.js:38:23900)
2025-02-06T22:10:53.6921276Z     at yI.runTo (/home/vsts/work/_tasks/CloudFormationCreateOrUpdateStack_7ef7cdfa-aa45-42c5-93c8-d7603643dd99/1.19.0/CloudFormationCreateOrUpdateStack.js:38:1153)
2025-02-06T22:10:53.6922454Z     at /home/vsts/work/_tasks/CloudFormationCreateOrUpdateStack_7ef7cdfa-aa45-42c5-93c8-d7603643dd99/1.19.0/CloudFormationCreateOrUpdateStack.js:38:1365
2025-02-06T22:10:53.6923324Z     at constructor.<anonymous> (/home/vsts/work/_tasks/CloudFormationCreateOrUpdateStack_7ef7cdfa-aa45-42c5-93c8-d7603643dd99/1.19.0/CloudFormationCreateOrUpdateStack.js:38:24115)
2025-02-06T22:10:53.6924343Z     at constructor.<anonymous> (/home/vsts/work/_tasks/CloudFormationCreateOrUpdateStack_7ef7cdfa-aa45-42c5-93c8-d7603643dd99/1.19.0/CloudFormationCreateOrUpdateStack.js:38:28399)
2025-02-06T22:10:53.6924832Z     at constructor.callListeners (/home/vsts/work/_tasks/CloudFormationCreateOrUpdateStack_7ef7cdfa-aa45-42c5-93c8-d7603643dd99/1.19.0/CloudFormationCreateOrUpdateStack.js:37:97445) {
2025-02-06T22:10:53.6925722Z   code: 'CredentialsError',
2025-02-06T22:10:53.6926570Z   time: 2025-02-06T22:10:53.688Z,
2025-02-06T22:10:53.6926859Z   requestId: 'REDACTED-BY-ZAFODB',
2025-02-06T22:10:53.6927277Z   statusCode: 403,
2025-02-06T22:10:53.6927663Z   retryable: false,
2025-02-06T22:10:53.6927904Z   retryDelay: 33.411946927934565,
2025-02-06T22:10:53.6928138Z   originalError: {
2025-02-06T22:10:53.6928374Z     message: 'Could not load credentials from constructor',
2025-02-06T22:10:53.6928634Z     code: 'CredentialsError',
2025-02-06T22:10:53.6928880Z     time: 2025-02-06T22:10:53.688Z,
2025-02-06T22:10:53.6929147Z     requestId: 'REDACTED-BY-ZAFODB',
2025-02-06T22:10:53.6929395Z     statusCode: 403,
2025-02-06T22:10:53.6929606Z     retryable: false,
2025-02-06T22:10:53.6929841Z     retryDelay: 33.411946927934565,
2025-02-06T22:10:53.6930074Z     originalError: {
2025-02-06T22:10:53.6930339Z       message: 'The security token included in the request is invalid.',
2025-02-06T22:10:53.6930591Z       code: 'InvalidClientTokenId',
2025-02-06T22:10:53.6930848Z       time: 2025-02-06T22:10:53.688Z,
2025-02-06T22:10:53.6931114Z       requestId: 'REDACTED-BY-ZAFODB',
2025-02-06T22:10:53.6931373Z       statusCode: 403,
2025-02-06T22:10:53.6931600Z       retryable: false,
2025-02-06T22:10:53.6931820Z       retryDelay: 33.411946927934565
2025-02-06T22:10:53.6932040Z     }
2025-02-06T22:10:53.6932229Z   }
2025-02-06T22:10:53.6932423Z }
2025-02-06T22:10:53.6936725Z ##[error]CredentialsError: Missing credentials in config, if using AWS_CONFIG_FILE, set AWS_SDK_LOAD_CONFIG=1
2025-02-06T22:10:53.7117519Z ##[section]Finishing: SAMPLE_TASK

Azure Pipeline log in case of success (~4 out of 5 cases):

Starting: SAMPLE_TASK
==============================================================================
Task         : AWS CloudFormation Create/Update Stack
Description  : Creates a new AWS CloudFormation stack or updates the stack if it exists.
Version      : 1.19.0
Author       : Amazon Web Services
Help         : Please refer to [AWS CloudFormation User Guide](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/) for more details on working with AWS CloudFormation.

More information on this task can be found in the [task reference](https://docs.aws.amazon.com/vsts/latest/userguide/cloudformation-create-update.html).

####Task Permissions
This task requires permissions to call the following AWS service APIs (depending on selected task options, not all APIs may be used):
* cloudformation:CreateChangeSet
* cloudformation:CreateStack
* cloudformation:DeleteChangeSet
* cloudformation:DescribeChangeSet
* cloudformation:DescribeStacks
* cloudformation:DescribeStackResources
* cloudformation:ExecuteChangeSet
* cloudformation:UpdateStack

The task may also require permissions to upload your application template to the specified Amazon S3 bucket.
==============================================================================
Configuring credentials for task
...configuring AWS credentials from service endpoint 'REDACTED-BY-ZAFODB'
Skipping Instance profile, we have OIDC enabled
...configuring AWS credentials from service endpoint 'REDACTED-BY-ZAFODB'
Getting OIDC Token...
OIDC Token generated: issuer: {https://vstoken.dev.azure.com/REDACTED-BY-ZAFODB } sub: {sc://REDACTED-BY-ZAFODB}, aud: {api://REDACTED-BY-ZAFODB}
Configuring region for task
...configured to use region eu-central-1, defined in task.
Assuming role via OIDC Token...
...role assumed via OIDC Token: arn:aws:sts::REDACTED-BY-ZAFODB
Configuring region for task
...configured to use region eu-central-1, defined in task.
Configuring credentials for task
...configuring AWS credentials from service endpoint 'REDACTED-BY-ZAFODB'
Skipping Instance profile, we have OIDC enabled
...configuring AWS credentials from service endpoint 'REDACTED-BY-ZAFODB'
Getting OIDC Token...
OIDC Token generated: issuer: {https://vstoken.dev.azure.com/REDACTED-BY-ZAFODB} sub: {sc://REDACTED-BY-ZAFODB}, aud: {api://REDACTED-BY-ZAFODB}
Configuring region for task
...configured to use region eu-central-1, defined in task.
Assuming role via OIDC Token...
...role assumed via OIDC Token: arn:aws:sts::REDACTED-BY-ZAFODB
Configuring region for task
...configured to use region eu-central-1, defined in task.
Checking existence for stack REDACTED-BY-ZAFODB
Stack exists, switching to update stack mode
Updating stack with template file /home/vsts/work/1/s/cloudformation/dms-source.yml
Loading template file from '/home/vsts/work/1/s/cloudformation/dms-source.yml'
Loading template parameters from task definition
Successfully loaded template parameters
Setting capability CAPABILITY_IAM for stack
Setting capability CAPABILITY_NAMED_IAM for stack
##[warning]WARNING: no changes were detected for the template or change set. The stack was not updated.
Skipping processing of stack outputs to build variables as task option is set to 'ignore' mode
Stack REDACTED-BY-ZAFODB processed successfully, stack ID arn:aws:cloudformation:eu-central-1:REDACTED-BY-ZAFODB
Finishing: SAMPLE_TASK

[HenrikStanley]

As a suggestion, a simple solution could be to implement some retry logic in case of an error or timeout.

If for whatever reason either the Agent, networking or AWS API does not respond, but it is temporary, a simple retry should fix it in the short term. It would be good to debug the root course of the timeouts, but regardless a retry would also help other temporary blibs that can occur in networked systems.

[hayemaxi]

Are you using microsoft-hosted agents?

We have noticed this issue appearing randomly the past few weeks in our test pipelines as well, although not nearly as commonly as 1 in 5. This most certainly appears to be some sort of underlying network issue. We use AZDO-provided APIs to handle fetching the OIDC token for us. However, we should certainly have retries for these cases. I will look into getting this added.