Request: Add way to only load a subset of channels and range of data

[akesner1]

Sorry for the post if I am missing something. I am wondering if this functionality could be added so we don't have to load an entire recording.

[jsiegle]

For the Binary and HDF5 formats (NWB and KWIK), the data is memory mapped and only loaded upon request. So this is already implemented for those formats.

For the Open Ephys format, the data is all loaded at once, so this is a feature we'll want to add. We don't have a timeline for this yet, but we'll leave this issue open because it's important to do.

[JonathanAMichaels]

I've also just realized that Open Ephys format files are loaded all at once. This is a huge issue, as all my recordings (not Neuropixels, but still large) are in that format and therefore cannot be loaded on most machines.

[jsiegle]

@medengineer we should add this feature ASAP.

Regarding the implementation, I think we should follow the conventions used by SpikeInterface:

For a Recording object, querying the .continuous property will still return all of the data as it does currently, but we will add a get_traces() method which can take start_frame, end_frame, and channels arguments, e.g.:

traces = recording.get_traces(start_frame=0, end_frame=30000, channels=np.arange(10))

would load the first 30k samples for the first 10 channels into memory.

[JonathanAMichaels]

Thanks @jsiegle. I'm not proud of it, but I've worked around this in the meantime by loading the data in matlab (while keeping things int16 as much as possible), then dumping to a binary.

[jsiegle]

This feature is implemented for the Open Ephys format in the latest version (now available to install via PyPI). See the open_ephys.analysis README file for details.

The Open Ephys format loader is only compatible with data saved by v0.6.x of the GUI...let us know if you need to load data saved by an earlier version, and we can port the code to the archive branch.