Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug] Out of memory when extracting large CSV from DCAS #389

Closed
danangmassandy opened this issue Jan 24, 2025 · 0 comments · Fixed by #395
Closed

[Bug] Out of memory when extracting large CSV from DCAS #389

danangmassandy opened this issue Jan 24, 2025 · 0 comments · Fixed by #395
Assignees
@danangmassandy
Labels
bug Something isn't working
Milestone
V2 - DCAS decisio...

Comments

@danangmassandy
Copy link
Collaborator

danangmassandy commented Jan 24, 2025 •
edited
Loading

From testing #380, I have run the pipeline and found out that duckdb failed to extract large CSV file and causes out of memory.

As an alternative solution, we could split into smaller csv files and ask Brian whether they are okay receiving multiple csv files. Once splitted, we could also merge them into 1 large csv file before sending into SFTP.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@danangmassandy danangmassandy

Labels
bug Something isn't working
Projects
None yet
Milestone
V2 - DCAS decision tree module operat...
Development

Successfully merging a pull request may close this issue.

set config for duckdb number of threads kartoza/tomorrownow_gap
1 participant
@danangmassandy