handle repeated 429s from Observable API with exponential back off #1761
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Framework already respects HTTP 429 rate limit responses from the Observable API. However, it will only retry once. This can sometimes not be enough, especially in cases where there are more than one Framework instances on the same IP, or Framework is not the only thing making request to the Observable API.
This PR changes the retry logic to retry up to 3 times, each time multiplying the Retry-After time the server instructed us to use by successive powers of two. After 3 times, Framework gives up and just lets the request fail.
In the worst case scenario, this would mean Framework waits in total 7 times longer than the server's "Retry-After" header. I've seen this Retry-After header as high as 60 seconds on our API, so this could mean a 7 minute delay.
I'm happy to bike shed on the number of retries or the way in which the time grows. We're also implementing some server side changes to help the situation.