Add stale time config to InstanceProfileCredentialsProvider

[L-Applin]

Add stale time config to InstanceProfileCredentialsProvider to allow for refreshing credentials earlier due to stale value. This will help prevent returning invalid credentials when an error is encountered during asynchronous refresh.

Motivation and Context

Fixes for #5247 , this will allow user to provide a higher value for prefetch to prevent returning expired credentails without changing default value. Increasing this value may lead to a higher rate of request to IMDS, so we ought to avoid changing the base default value for it.

Modifications

• Added the staleTime configuration option to InstanceProfileCredentialsProvider
• Added retry strategy for InstanceProfileCredentialsProvider

Testing

Added unit tests

Screenshots (if appropriate)

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)

[steveloughran]

you need to change that default to be > 10s; without that any 503 response is not going be recovered from.

Those of us who have encountered the problem will know to explicitly change it, but people who haven't yet hit it will only find out when things go wrong.

There is also the subtlety that all processes running in the same VM will be using the same timeout, as it is based on when the shared credentials expire. The more processes running, the more risk of that 503 -regardless of what the value is. This means that any constant value can create a Thundering Herd.

The "extra load" concern is marginal, given refresh time is every hour. spreading out that load is likely to be better as any herd-triggered retry is load in itself.

I'd suggest some larger number and maybe a jitter of a few seconds to spread out load from many processes.

[L-Applin]

you need to change that default to be > 10s; without that any 503 response is not going be recovered from.

Those of us who have encountered the problem will know to explicitly change it, but people who haven't yet hit it will only find out when things go wrong.

There is also the subtlety that all processes running in the same VM will be using the same timeout, as it is based on when the shared credentials expire. The more processes running, the more risk of that 503 -regardless of what the value is. This means that any constant value can create a Thundering Herd.

The "extra load" concern is marginal, given refresh time is every hour. spreading out that load is likely to be better as any herd-triggered retry is load in itself.

I'd suggest some larger number and maybe a jitter of a few seconds to spread out load from many processes.

There is already jitter, up to 1 minute before expiration time. Also, I am weary of changing default values, it may cause unexpected behaviour change in user applications and we try to avoid those. Is the ability to configure the value manually on the builder yourself not a viable solution for your use case?

[steveloughran]

I'm happy with the change -but do think you should consider documenting this for others

[L-Applin]

I'm happy with the change -but do think you should consider documenting this for others

Good point, I added a note about the issue in the javadoc for staleTime

[sonarqubecloud]

Quality Gate failed

Failed conditions
73.3% Coverage on New Code (required ≥ 80%)

See analysis details on SonarQube Cloud