Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Explore other cache options with Fsspec #27

Open
CodyCBakerPhD opened this issue Feb 20, 2024 · 1 comment
Open

Explore other cache options with Fsspec #27

CodyCBakerPhD opened this issue Feb 20, 2024 · 1 comment
Assignees

Comments

@CodyCBakerPhD
Copy link
Collaborator

Fsspec offers a large number of cache options

Our current test is only for the non-cache behavior

PyNWB has a tutorial for the deafult/basic options: https://pynwb.readthedocs.io/en/stable/tutorials/advanced_io/streaming.html#streaming-method-1-fsspec

But fsspec also has MANY other cache options and read-ahead styles: https://github.com/fsspec/filesystem_spec/blob/master/fsspec/caching.py#L72

We should push those to their limits in our tests to see what works best (especially in cases of repeated access to contents)

@oruebel
Copy link
Contributor

oruebel commented Apr 24, 2024

PyNWB has a tutorial for the deafult/basic options: https://pynwb.readthedocs.io/en/stable/tutorials/advanced_io/streaming.html#streaming-method-1-fsspec

#43 implements this, but it does not implement other caching options for fsspec.

@oruebel oruebel changed the title Add FsspecWithCache option Explore other cache options with Fsspec Apr 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants