Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Address size limitations #3

Open
kelle opened this issue Feb 7, 2022 · 0 comments
Open

Address size limitations #3

kelle opened this issue Feb 7, 2022 · 0 comments

Comments

@kelle
Copy link
Contributor

kelle commented Feb 7, 2022

"The paper suggests that SQLite and astrodbkit can be scaled to arbitrary size and scope (in the SQLite file size up to 140TB and large number of collaborations). The workflow as described will hit pratical limits much sooner than this. It's fine for a very small SQLite file to be directly included in a repository, but committing hundreds of megabytes or more as a binary blob will dramatically increase the size of the repository each time data is added - not incrementally, but by the size of the full database each time. The practical limit is much smaller. Further (as described below), the workflow only works when one user makes updates to the database at one time. The chances of "collisions" when multiple users do this at the same time increases quickly as collaborators are added. Of course, many of these problems are solved by using a client/server database where the data is not located in the repository."

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant