Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reduce memory usage: zero-copy #62

Open
aufdenkampe opened this issue Dec 21, 2023 · 1 comment
Open

Reduce memory usage: zero-copy #62

aufdenkampe opened this issue Dec 21, 2023 · 1 comment
Labels
enhancement New feature or request refactor-core For core functionality, not model specifc

Comments

@aufdenkampe
Copy link
Member

aufdenkampe commented Dec 21, 2023

In @sjordan29's exploration of performance issues in coupling TSM to a test case of ClearWater-riverine, we suspected that we might be running out of memory. This corresponded to our performance profiling findings described in #57 (comment) where a substantial amount of time was spent in xarray merge/copy/init functions.

Zero-copy

We want to move as much as possible to a zero-copy approach when running (and coupling) these models. Remember that xarray.DataArray is really just a numpy.ndarray under the hood, so we should use pointers to that single source of data (rather than copying) as much as possible.

Zero-copy approaches complement the pre-initialized array approach described in:

@aufdenkampe aufdenkampe added enhancement New feature or request refactor-core For core functionality, not model specifc labels Dec 21, 2023
@aufdenkampe aufdenkampe changed the title Reduce memory usage: zero-copy and Dask.array Reduce memory usage: zero-copy Feb 26, 2024
@aufdenkampe
Copy link
Member Author

I just edited this issue title and description to split out the following into separate issues:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request refactor-core For core functionality, not model specifc
Projects
None yet
Development

No branches or pull requests

1 participant