Fix encodeBase64 not handling uint8 arrays beyond ~100kB #150
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation: When I was trying to upload bigger data to polybase it was failing with a
RangeError: Maximum call stack size exceeded
which turned out to be caused by theencodeBase64()
function which apparently wasn't able to handle data bigger than ~100kB.This PR reimplements the
encodeBase64()
in a safer way which works with bigger arrays without issues, even beyond 1MB - not sure if there's an upper bound (in practice I expect that code to be too slow/blocking for files larger than several MB), I guess a proper solution for bigger data would be to stream/chunk it somehow which would require designing a whole new API call, nevertheless, this solution is an improvement respective to the current implementation and it was already enough for my usecasemore info on the issue e.g. here: mathiasbynens/base64#13
How to test: Try creating a colletion with a buffer field containing a data item of more than 100kB: