WithRetries Not Waiting Long Enough

[GusPrice]

Checklist

• I have looked into the README and have not found a suitable solution or answer.
• I have looked into the documentation and have not found a suitable solution or answer.
• I have searched the issues and have not found a suitable solution or answer.
• I have upgraded to the latest version of this SDK and the issue still persists.
• I have searched the Auth0 Community forums and have not found a suitable solution or answer.
• I agree to the terms within the Auth0 Code of Conduct.

Description

I am running into an issue where when I receive a 429 response from the management api, it seems the retries are not properly waiting to try again.

It seems that with this retry option, or just using the default retries I frequently run into an issue where the client gets a 429, but retries too soon. It seems like it's using the header value and getting to the final return in backoffDelay() in internal/client/client.go:

return time.Duration(resetAt-time.Now().Unix()) * time.Second

But it seems like maybe this math isn't quite waiting long enough for things to have reset. Here's an example of the debug logs where you can see the reset header value is the current time the code is running at but it continues to fail.

Sample Output

2025/02/06 11:00:31 
HTTP/2.0 429 Too Many Requests
Content-Length: 120
Access-Control-Expose-Headers: WWW-Authenticate,Server-Authorization
Age: 1
Alt-Svc: h3=":443"; ma=86400
Cache-Control: public, max-age=1
Cf-Cache-Status: HIT
Cf-Ray: 90dd62320973645e-SJC
Content-Type: application/json; charset=utf-8
Date: Thu, 06 Feb 2025 19:00:31 GMT
Retry-After: 1
Server: cloudflare
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: origin, Accept-Encoding
X-Auth0-L: 0.025
X-Content-Type-Options: nosniff
X-Ratelimit-Limit: 10
X-Ratelimit-Remaining: 0
X-Ratelimit-Reset: 1738868431

{"statusCode":429,"error":"Too Many Requests","message":"Global limit has been reached","errorCode":"too_many_requests"}

2025/02/06 11:00:31 
GET /api/v2/users-by-email?email=test%gmail.com HTTP/1.1
Host: melkor-dev.us.auth0.com
User-Agent: Go-Auth0/1.15.0
Auth0-Client: eyJuYW1lIjoiZ28tYXV0aDAiLCJ2ZXJzaW9uIjoiMS4xNS4wIiwiZW52Ijp7ImdvIjoiZ28xLjIyLjQifX0=
Content-Type: application/json
Accept-Encoding: gzip


2025/02/06 11:00:31 
HTTP/2.0 429 Too Many Requests
Content-Length: 120
Access-Control-Expose-Headers: WWW-Authenticate,Server-Authorization
Age: 1
Alt-Svc: h3=":443"; ma=86400
Cache-Control: public, max-age=1
Cf-Cache-Status: HIT
Cf-Ray: 90dd623269d7645e-SJC
Content-Type: application/json; charset=utf-8
Date: Thu, 06 Feb 2025 19:00:31 GMT
Retry-After: 1
Server: cloudflare
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: origin, Accept-Encoding
X-Auth0-L: 0.025
X-Content-Type-Options: nosniff
X-Ratelimit-Limit: 10
X-Ratelimit-Remaining: 0
X-Ratelimit-Reset: 1738868431

{"statusCode":429,"error":"Too Many Requests","message":"Global limit has been reached","errorCode":"too_many_requests"}

2025/02/06 11:00:31 
GET /api/v2/users-by-email?email=test%gmail.com HTTP/1.1
Host: melkor-dev.us.auth0.com
User-Agent: Go-Auth0/1.15.0
Auth0-Client: eyJuYW1lIjoiZ28tYXV0aDAiLCJ2ZXJzaW9uIjoiMS4xNS4wIiwiZW52Ijp7ImdvIjoiZ28xLjIyLjQifX0=
Content-Type: application/json
Accept-Encoding: gzip


2025/02/06 11:00:31 
HTTP/2.0 429 Too Many Requests
Content-Length: 120
Access-Control-Expose-Headers: WWW-Authenticate,Server-Authorization
Age: 1
Alt-Svc: h3=":443"; ma=86400
Cache-Control: public, max-age=1
Cf-Cache-Status: HIT
Cf-Ray: 90dd6232ba12645e-SJC
Content-Type: application/json; charset=utf-8
Date: Thu, 06 Feb 2025 19:00:31 GMT
Retry-After: 1
Server: cloudflare
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: origin, Accept-Encoding
X-Auth0-L: 0.025
X-Content-Type-Options: nosniff
X-Ratelimit-Limit: 10
X-Ratelimit-Remaining: 0
X-Ratelimit-Reset: 1738868431

{"statusCode":429,"error":"Too Many Requests","message":"Global limit has been reached","errorCode":"too_many_requests"}

2025/02/06 11:00:31 
GET /api/v2/users-by-email?email=test%gmail.com HTTP/1.1
Host: melkor-dev.us.auth0.com
User-Agent: Go-Auth0/1.15.0
Auth0-Client: eyJuYW1lIjoiZ28tYXV0aDAiLCJ2ZXJzaW9uIjoiMS4xNS4wIiwiZW52Ijp7ImdvIjoiZ28xLjIyLjQifX0=
Content-Type: application/json
Accept-Encoding: gzip


2025/02/06 11:00:31 
HTTP/2.0 429 Too Many Requests
Content-Length: 120
Access-Control-Expose-Headers: WWW-Authenticate,Server-Authorization
Age: 1
Alt-Svc: h3=":443"; ma=86400
Cache-Control: public, max-age=1
Cf-Cache-Status: HIT
Cf-Ray: 90dd62330a69645e-SJC
Content-Type: application/json; charset=utf-8
Date: Thu, 06 Feb 2025 19:00:31 GMT
Retry-After: 1
Server: cloudflare
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: origin, Accept-Encoding
X-Auth0-L: 0.025
X-Content-Type-Options: nosniff
X-Ratelimit-Limit: 10
X-Ratelimit-Remaining: 0
X-Ratelimit-Reset: 1738868431

{"statusCode":429,"error":"Too Many Requests","message":"Global limit has been reached","errorCode":"too_many_requests"}

2025/02/06 11:00:31 
GET /api/v2/users-by-email?email=test%gmail.com HTTP/1.1
Host: melkor-dev.us.auth0.com
User-Agent: Go-Auth0/1.15.0
Auth0-Client: eyJuYW1lIjoiZ28tYXV0aDAiLCJ2ZXJzaW9uIjoiMS4xNS4wIiwiZW52Ijp7ImdvIjoiZ28xLjIyLjQifX0=
Content-Type: application/json
Accept-Encoding: gzip


2025/02/06 11:00:31 
HTTP/2.0 429 Too Many Requests
Content-Length: 120
Access-Control-Expose-Headers: WWW-Authenticate,Server-Authorization
Age: 1
Alt-Svc: h3=":443"; ma=86400
Cache-Control: public, max-age=1
Cf-Cache-Status: HIT
Cf-Ray: 90dd62336aa3645e-SJC
Content-Type: application/json; charset=utf-8
Date: Thu, 06 Feb 2025 19:00:31 GMT
Retry-After: 1
Server: cloudflare
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: origin, Accept-Encoding
X-Auth0-L: 0.025
X-Content-Type-Options: nosniff
X-Ratelimit-Limit: 10
X-Ratelimit-Remaining: 0
X-Ratelimit-Reset: 1738868431

{"statusCode":429,"error":"Too Many Requests","message":"Global limit has been reached","errorCode":"too_many_requests"}

Expectation

I would expect that 1 retry, 2 maximum would wait long enough to avoid this 429 burst limit of 10 that is specific to my ip address. Currently it seems to try again too soon relative the reset value returned by the header.

Reproduction

I am running something similar to the follow example code (simplified):

	auth0Management, err := a0mgmt.New(
		cfg.Auth0.ManagementDomain,
		a0mgmt.WithClientCredentials(ctx, cfg.Auth0.ClientID, cfg.Auth0.ClientSecret),
		a0mgmt.WithDebug(true),
		a0mgmt.WithRetries(5, []int{http.StatusTooManyRequests}),
	)
	if err != nil {
		log.Fatalf("failed to initialize the auth0 management API client: %+v", err)
	}

	emails := []string{"[email protected]", "[email protected]", "[email protected]", "[email protected]", "[email protected]", "[email protected]", "[email protected]", "[email protected]", "[email protected]", "[email protected]", "[email protected]", "[email protected]", "[email protected]", "[email protected]", "[email protected]", "[email protected]", "[email protected]", "[email protected]", "[email protected]", "[email protected]"}
	for _, email := range emails {
		users, err := a0.User.ListByEmail(ctx, email)
		if err != nil {
			return &a0mgmt.User{}, err
		}
		fmt.Println(users)
	}

Auth0 Go SDK version

1.16.0

[GusPrice]

Per Alvaro with Auth0 support:

I think the problem is with the GO SDK and how it handles these burst rate limit resets. Here is what is happening in the code:
1. The code calculates a wait time using exponential backoff, the delay is supposed to double with each retry attempt.
2. A random time is added to the delay in a range between the min and max values.
3. The function checks if the response status is 429. If no 429 then returns the wait time.
4. When the status is 429 it looks at the x-ratelimit-reset header and attempts to parse this value into resetAt.
5. If parsing fails, then it returns the wait time.
6. If the resetAt timestamp is greater than the current time plus the maximum delay, it returns the wait time.
7. If the rate limit reset is within the max delay, it returns the remaining time until the reset occurs (resetAt - time.now().unix()), converts this to seconds, and then to time.duration. 
 
The problem is happening because there is no difference between the resetAt value and the current time. In the example you provided, the resetAt timestamp is 1738868431 (2025/02/06 11:00:31) and the current time is also 2025/02/06 11:00:31, you would need to add a delay for this to work as expected. This retry logic does work for the 429 responses from the global Management API limit. That refill rate is in minutes so each new request is added in seconds instead of milliseconds.

[developerkunal]

Hi @GusPrice,

I hope you're doing well!

I tried to investigate this issue, but I wasn’t able to find any problems. Could you let me know if you're using a free tenant or an enterprise one?

Thanks!

[GusPrice]

Hi @developerkunal,

That's unfortunate. We are on an enterprise account; but the tenant is a development level.

[GusPrice]

@developerkunal just following up here to see if you were able to look any further?

[developerkunal]

Hi @GusPrice,

We are currently looking into this! This issue may not be limited to the current SDK but could also affect other SDKs.