Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix encodeBase64 not handling uint8 arrays beyond ~100kB #150

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

refi93
Copy link

@refi93 refi93 commented Jul 19, 2023

Motivation: When I was trying to upload bigger data to polybase it was failing with a RangeError: Maximum call stack size exceeded which turned out to be caused by the encodeBase64() function which apparently wasn't able to handle data bigger than ~100kB.

This PR reimplements the encodeBase64() in a safer way which works with bigger arrays without issues, even beyond 1MB - not sure if there's an upper bound (in practice I expect that code to be too slow/blocking for files larger than several MB), I guess a proper solution for bigger data would be to stream/chunk it somehow which would require designing a whole new API call, nevertheless, this solution is an improvement respective to the current implementation and it was already enough for my usecase

more info on the issue e.g. here: mathiasbynens/base64#13

How to test: Try creating a colletion with a buffer field containing a data item of more than 100kB:

const pb = new Polybase({
  defaultNamespace: 'mynamespace',
})
await pb
  .collection('MyCollection')
  .create([Array(1_000_000).fill(0)]) // on main this would fail with "RangeError: Maximum call stack size exceeded"

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant