[SVCS-899] Remove cute trick to deal with buffer size #359
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Ticket
https://openscience.atlassian.net/browse/SVCS-899
Purpose
Entire files were being pulled into memory when the client's connection to waterbutler was faster than waterbutler's connection to the provider. This is less than ideal. Wrapping around a socket is cute, but it's not really the right way to do it. Put an actual limit on the buffer size and wait for it to get written. Sending it through a unix socket was a quick and dirty way to get some control into the tornado's
data_received
handler method, letting us wait until some of the data had been read; The StreamReader has some functionality built in around limits to the buffer size, but if it can't pause the coroutine that's writing things to it, it can't do anything with the data other than save it to it's buffer anyway, otherwise it loses data.The RequestStreamReader can be tweaked a bit to handle writes to its buffer with a future, which would let it achieve the same thing without needing to wrap a streamreader in another streamreader, and without needing to open up a socketr pair.
The commit that added this trick: bb77d79
Changes
Side effects
QA Notes
Deployment Notes