[SVCS-894] Cache File During Contiguous Upload For Dropbox #368
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
cslzchen
commented
Oct 9, 2018
Dropbox's rate limiting feature may cause uploading/copying/moving many files in parallel to fail with either "too_many_requests" or "too_many_write_operations". The former is literally used for "too many requests" while the latter for namespace lock contentions. See https://www.dropbox.com/developers/reference/data-ingress-guide for more details. In addition, Dropbox doesn't reveal how they rate-limit requests and is actively testing different algorithms. They recommend clients to retry according to the "Retry-After" header in the 429 response. However, the retry doesn't work in a straightforward way for upload requests since the stream will have been consumed when the request is finished. Please note both inter copy and move use upload internally. The solution is to cache the stream locally into a temporary file and stream from it for both the initial request and following 429 retries.
During local testing, all 429 failures succeeded upon first retry. The issue turned out to be "namespace lock contention" instead of "two many requests". It is reasonable to set the default maximum value to 2 retires per upload.
cslzchen
force-pushed
the
fix/dropbox-too-many-write-ops
branch
from
550c649 to
a214e84
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Ticket
https://openscience.atlassian.net/browse/SVCS-894
Purpose
Dropbox's rate limit feature may cause uploading/copying/moving many files to fail with "too_many_requests" & "too_many_write_operations". The former is for too many requests literally and the former is for namespace lock contentions. Unlike GitHub, Dropbox doesn't provide detailed information on how they rate limit requests but simply asks clients to retry according to the "Retry-After" header in the 429 response.
The retry doesn't work in a straightforward way for upload (inter copy and move use upload internally) since the stream will have been consumed when the request is finished. The solution is to cache the stream locally into a temporary file and stream from the file for both the initial request and following 429 retries.
Changes
TBD
Side effects
TBD
QA Notes
TBD
Deployment Notes
No