Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make the default number of partions in json-file something larger than 1 #136

Open
ingomueller-net opened this issue Nov 28, 2018 · 2 comments

Comments

@ingomueller-net
Copy link
Contributor

If you say:

json-file("...")

the computation will run with only a single partition, doing the scan inside a single executor. A much better default value would be the number of executors, or 10 times that, or simply 100, IMO.

@ghislainfourny
Copy link
Member

Agreed. 100 would be a better default number.

@ghislainfourny
Copy link
Member

The workaround is:

json-file("...", 100)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants