Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DCAS Testing - Ensure performance for processing 5M Farms #380

Open
danangmassandy opened this issue Jan 22, 2025 · 1 comment
Open

DCAS Testing - Ensure performance for processing 5M Farms #380

danangmassandy opened this issue Jan 22, 2025 · 1 comment
Assignees
@danangmassandy
Milestone
V2 - DCAS decisio...

Comments

@danangmassandy
Copy link
Collaborator

danangmassandy commented Jan 22, 2025 •
edited
Loading

Ensure the task can run with memory under 4GB and good cpu usage. By this ticket, we should have the estimation to determine the number of partitions to process 5M FarmRegistry.
@danangmassandy danangmassandy self-assigned this Jan 22, 2025
This was referenced Jan 23, 2025
Merged
Closed
@danangmassandy
Copy link
Collaborator Author

The process is able to run with maximum memory 4 GB and total time is around 2 hours 20 minutes to build the geoparquet for Farm Registry. Another note is to set a configuration to number of threads of DuckDB is using so we can control the CPU usage.
Currently, the process extracting the csv file is failed because out of memory and I've created a new ticket to address this issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@danangmassandy danangmassandy

Labels
None yet
Projects
None yet
Milestone
V2 - DCAS decision tree module operat...
Development

No branches or pull requests
1 participant
@danangmassandy