Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Write to Zarr store using h5py-like interface #28

Merged
merged 42 commits into from
Apr 4, 2024
Merged

Write to Zarr store using h5py-like interface #28

merged 42 commits into from
Apr 4, 2024

Conversation

magland
Copy link
Collaborator

@magland magland commented Mar 25, 2024

@codecov-commenter
Copy link

Welcome to Codecov 🎉

Once you merge this PR into your default branch, you're all set! Codecov will compare coverage reports and display results in all future pull requests.

Thanks for integrating Codecov - We've got you covered ☂️

@oruebel
Copy link

oruebel commented Mar 25, 2024

Just to clarify, if I understand this correctly, lindi.LindiH5pyFile is here backed by a Zarr store, i.e., it is representing a Zarr file as if it where h5py and as such, pynwb.NWBHDF5IO is here actually writing to Zarr. I.e., in some sense this would be in a form a replacement forZarrIO in hdmf_zarr.

store = zarr.DirectoryStore(zarr_dirname)
zarr.group(store=store) # create a root group
with lindi.LindiH5pyFile.from_zarr_store(store, mode='a') as client:
with pynwb.NWBHDF5IO(file=client, mode='r+') as io:
io.write(nwbfile) # type: ignore

@magland
Copy link
Collaborator Author

magland commented Mar 25, 2024

Just to clarify, if I understand this correctly, lindi.LindiH5pyFile is here backed by a Zarr store, i.e., it is representing a Zarr file as if it where h5py and as such, pynwb.NWBHDF5IO is here actually writing to Zarr. I.e., in some sense this would be in a form a replacement forZarrIO in hdmf_zarr.

store = zarr.DirectoryStore(zarr_dirname)
zarr.group(store=store) # create a root group
with lindi.LindiH5pyFile.from_zarr_store(store, mode='a') as client:
with pynwb.NWBHDF5IO(file=client, mode='r+') as io:
io.write(nwbfile) # type: ignore

@oruebel Yes that's right.

tests/test_remote_data.py Outdated Show resolved Hide resolved
tests/test_zarr_write.py Outdated Show resolved Hide resolved
tests/test_copy.py Outdated Show resolved Hide resolved
@rly
Copy link
Contributor

rly commented Apr 4, 2024

The suggested changes around np.float128/np.complex256 and the change of mode="a" to mode="r+" were necessary to make examples/example_create_zarr_nwb.py run on my machine.

@rly
Copy link
Contributor

rly commented Apr 4, 2024

I did not do heavy testing of this PR, but I read through it all, and as far as I can tell, it makes sense and looks good. Nicely done!

HDMF supports a lot of weird edge cases. I'll see if I can think up more tests to what you already have here.

@magland
Copy link
Collaborator Author

magland commented Apr 4, 2024

I did not do heavy testing of this PR, but I read through it all, and as far as I can tell, it makes sense and looks good. Nicely done!

HDMF supports a lot of weird edge cases. I'll see if I can think up more tests to what you already have here.

Thanks Ryan!

@magland magland merged commit b2c2788 into main Apr 4, 2024
6 checks passed
@magland magland deleted the write branch April 4, 2024 18:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants