Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dat_exploded csv files are too large for GitHub #147

Open
df511 opened this issue Jun 8, 2021 · 0 comments
Open

Dat_exploded csv files are too large for GitHub #147

df511 opened this issue Jun 8, 2021 · 0 comments

Comments

@df511
Copy link
Contributor

df511 commented Jun 8, 2021

dat_exploded gained a considerable amount of data this year, as we are using unfiltered data for map-making and added three new regions and a new year's data. For this reason, the files needed to be parsed to by region to be exported. However, even these regional files are too large as gzipped CSVs (csv.gz) (many of these files are >100 MB, GitHub's file size limit).

For this reason, this year's data_exploded files were exported as .rds

The Gulf of Alaska dat_exploded (dat_explodedGulf of Alaska.rds.zip) needed to be additionally compressed to be just under 100 MB.

We need to consider other compression and/or parsing options for next year.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant