Occasional 429 errors (Too Many Requests) in Drive activity polls #2161
Comments
jpd236
added a commit
to jpd236/jolly-roger
that referenced
this issue
Jun 28, 2024
The default retry policy (documented [here](https://github.com/googleapis/gaxios/#request-options)) does not retry POST requests. But most Google API calls use POST requests under the covers, including read-only APIs like [Drive Activity](https://developers.google.com/drive/activity/v2/reference/rest/v2/activity/query). So we can also ask to retry POST requests. We retain other aspects of the default policy, such as the number of attempts (3) and the HTTP status codes to retry (100-109, 408, 429, 5xx). If a request succeeds on a retry, the failure is kept silent and does not pollute the logs. The main risk is that a POST method might mutate some state despite returning a failure, and that inline retries might handle that incorrectly or otherwise make things worse than retrying the full flow from the ground up. This doesn't seem likely to be an issue with the current set of API calls, but if desired, we could narrow this change to specific known-safe requests. See deathandmayhem#2160 See deathandmayhem#2161
jpd236
added a commit
to jpd236/jolly-roger
that referenced
this issue
Jun 28, 2024
The default retry policy (documented [here](https://github.com/googleapis/gaxios/#request-options)) does not retry POST requests. But most Google API calls use POST requests under the covers, including read-only APIs like [Drive Activity](https://developers.google.com/drive/activity/v2/reference/rest/v2/activity/query). So we can also ask to retry POST requests. We retain other aspects of the default policy, such as the number of attempts (3) and the HTTP status codes to retry (100-109, 408, 429, 5xx). If a request succeeds on a retry, the failure is kept silent and does not pollute the logs. The main risk is that a POST method might mutate some state despite returning a failure, and that inline retries might handle that incorrectly or otherwise make things worse than retrying the full flow from the ground up. This doesn't seem likely to be an issue with the current set of API calls, but if desired, we could narrow this change to specific known-safe requests. See deathandmayhem#2160 See deathandmayhem#2161
jpd236
added a commit
to jpd236/jolly-roger
that referenced
this issue
Jun 28, 2024
The default retry policy (documented [here](https://github.com/googleapis/gaxios/#request-options)) does not retry POST requests. But most Google API calls use POST requests under the covers, including read-only APIs like [Drive Activity](https://developers.google.com/drive/activity/v2/reference/rest/v2/activity/query). So we can also ask to retry POST requests. We retain other aspects of the default policy, such as the number of attempts (3) and the HTTP status codes to retry (100-109, 408, 429, 5xx). If a request succeeds on a retry, the failure is kept silent and does not pollute the logs. The main risk is that a POST method might mutate some state despite returning a failure, and that inline retries might handle that incorrectly or otherwise make things worse than retrying the full flow from the ground up. This doesn't seem likely to be an issue with the current set of API calls, but if desired, we could narrow this change to specific known-safe requests. See deathandmayhem#2160 See deathandmayhem#2161
jpd236
added a commit
to jpd236/jolly-roger
that referenced
this issue
Jun 28, 2024
The default retry policy (documented [here](https://github.com/googleapis/gaxios/#request-options)) does not retry POST requests. But most Google API calls use POST requests under the covers, including read-only APIs like [Drive Activity](https://developers.google.com/drive/activity/v2/reference/rest/v2/activity/query). So we can also ask to retry POST requests. We retain other aspects of the default policy, such as the number of attempts (3) and the HTTP status codes to retry (100-109, 408, 429, 5xx). If a request succeeds on a retry, the failure is kept silent and does not pollute the logs. The main risk is that a POST method might mutate some state despite returning a failure, and that inline retries might handle that incorrectly or otherwise make things worse than retrying the full flow from the ground up. This doesn't seem likely to be an issue with the current set of API calls, but if desired, we could narrow this change to specific known-safe requests. See deathandmayhem#2160 See deathandmayhem#2161
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
My server logs show occasional instances of 429s in the Drive activity queries:
The overall error rate looks quite low in the console (0.02%), though it is a bit noisy. From what I can tell we're doing between 10-15 queries per minute, and the limit per minute per user is 100.
It seems like there might be some retries here already; I wonder if we just need to back off a bit more in this case.
The text was updated successfully, but these errors were encountered: