Error running LINDI in a subproccess

[oruebel]

@magland @rly we are working on the nwb-benchmarks and am encountering a strange issue NeurodataWithoutBorders/nwb_benchmarks#57 Do you have an idea? It looks like this might be related to running LINDI in a subprocess. The particular benchmark loads data from a remote file by slicing into a LINDI dataset. Simply opening a file with LINDI (incl. read with PyNWB) seems to work. The relevant code in the benchmark suite is: https://github.com/NeurodataWithoutBorders/nwb_benchmarks/blob/main/src/nwb_benchmarks/benchmarks/time_remote_slicing.py#L139-L156 which also uses these helper functions to open the file https://github.com/NeurodataWithoutBorders/nwb_benchmarks/blob/fb464f1a7ab6a2c4b4ce91b798ac68b088cf514d/src/nwb_benchmarks/core/_streaming.py#L205-L226

[magland]

In this case, does the s3_url point to an hdf5 file in the bucket? If so, that's not going to work... you need to point to the corresponding prepared json file. If it's a public dandi file, I can provide a formula for determining the proper url.

[oruebel]

The parameters are:

'''
lindi_remote_rfs_parameter_cases = dict(
# TODO: Just an example case for testing. Replace with real test case
BaseExample=dict(
s3_url="https://kerchunk.neurosift.org/dandi/dandisets/000939/assets/11f512ba-5bcf-4230-a8cb-dc8d36db38cb/zarr.json",
object_name="accelerometer",
slice_range=(slice(0, 30_000), slice(0, 3)),
),
)
'''

I.e., for now I am just using one of the examples from the README to check that the benchmarks are working. We are generating LINDI files for the actual test datasets we are planning to use. I used the version of lindi from PiPy. Let me know in case I should use a different file or version of lindi.

It looks like the read itself is working, ie, we are getting back an array with the expected shape from the slicing. It looks like the error is occurring in the teardown afterwards when the subprocess is being cleaned up.

[magland]

I am not able to reproduce. See below. I am on Ubuntu, and I installed nwb_benchmarks in a conda environment as in the docs. What OS are you using?

$ nwb_benchmarks run --bench time_remote_slicing.LindiFileReadRemoteReferenceFileSystemContinuousSliceBenchmark --debug
· Running '/home/magland/miniconda3/envs/nwb_benchmarks/bin/python -c import sys; print(str(sys.version_info[0]) + "." + str(sys.version_info[1]))'
  OUTPUT -------->
  3.11
· Running '/home/magland/miniconda3/envs/nwb_benchmarks/bin/git config init.defaultBranch'
  OUTPUT -------->
  main
· Running '/home/magland/miniconda3/envs/nwb_benchmarks/bin/git rev-parse 64cc427^{commit}'
  OUTPUT -------->
  64cc427d04423c5ce7ec93ca535d58e59cbd45f8
· Running '/home/magland/miniconda3/envs/nwb_benchmarks/bin/git rev-parse main^{commit}'
  OUTPUT -------->
  64cc427d04423c5ce7ec93ca535d58e59cbd45f8
· Discovering benchmarks
·· Running '/home/magland/miniconda3/envs/nwb_benchmarks/bin/git rev-parse main^{commit}'
   OUTPUT -------->
   64cc427d04423c5ce7ec93ca535d58e59cbd45f8
·· Running '/home/magland/miniconda3/envs/nwb_benchmarks/lib/python3.11/site-packages/asv/benchmark.py discover /home/magland/src/nwb_benchmarks/src/nwb_benchmarks/benchmarks /tmp/tmpe540k9g8/result.json' in existing-py_home_magland_miniconda3_envs_nwb_benchmarks_bin_python
·· Running '/home/magland/miniconda3/envs/nwb_benchmarks/bin/python /home/magland/miniconda3/envs/nwb_benchmarks/lib/python3.11/site-packages/asv/benchmark.py discover /home/magland/src/nwb_benchmarks/src/nwb_benchmarks/benchmarks /tmp/tmpe540k9g8/result.json'
· Running 1 total benchmarks (1 commits * 1 environments * 1 benchmarks)
[ 0.00%] · Running '/home/magland/miniconda3/envs/nwb_benchmarks/bin/git name-rev --name-only --exclude=remotes/* --no-undefined 64cc427d04423c5ce7ec93ca535d58e59cbd45f8'
           OUTPUT -------->
           main
[ 0.00%] · For nwb_benchmarks commit 64cc427d <main>:
[ 0.00%] ·· Building for existing-py_home_magland_miniconda3_envs_nwb_benchmarks_bin_python
[ 0.00%] ·· Running '/home/magland/miniconda3/envs/nwb_benchmarks/bin/git rev-list -n 1 --format=%at 64cc427d04423c5ce7ec93ca535d58e59cbd45f8'
            OUTPUT -------->
            commit 64cc427d04423c5ce7ec93ca535d58e59cbd45f8
            1714497072
[ 0.00%] ·· Benchmarking existing-py_home_magland_miniconda3_envs_nwb_benchmarks_bin_python
[ 0.00%] ··· Running '/home/magland/miniconda3/envs/nwb_benchmarks/lib/python3.11/site-packages/asv/benchmark.py run_server /home/magland/src/nwb_benchmarks/src/nwb_benchmarks/benchmarks /tmp/asv-forkserver-9fj1bwct/socket' in existing-py_home_magland_miniconda3_envs_nwb_benchmarks_bin_python
[ 0.00%] ··· Running '/home/magland/miniconda3/envs/nwb_benchmarks/bin/python /home/magland/miniconda3/envs/nwb_benchmarks/lib/python3.11/site-packages/asv/benchmark.py run_server /home/magland/src/nwb_benchmarks/src/nwb_benchmarks/benchmarks /tmp/asv-forkserver-9fj1bwct/socket'
[100.00%] ··· ...ileSystemContinuousSliceBenchmark.time_slice                 ok
[100.00%] ··· ============================================================================================================= =============== ============================================ ============
                                                                  s3_url                                                      object_name                   slice_range                              
              ------------------------------------------------------------------------------------------------------------- --------------- -------------------------------------------- ------------
               https://kerchunk.neurosift.org/dandi/dandisets/000939/assets/11f512ba-5bcf-4230-a8cb-dc8d36db38cb/zarr.json   accelerometer   (slice(0, 30000, None), slice(0, 3, None))   1.97±0.2ms 
              ============================================================================================================= =============== ============================================ ============

tag @CodyCBakerPhD

EDIT: I also tried this on github codespaces and did not see any error

[oruebel]

What OS are you using?

I am running on MacOS 12.5.1 (monterey) with a M1 Max chip

[CodyCBakerPhD]

Also replicated on Windows 11, but I'll see if I can't make a minimal example and get it in a CI

[magland]

Unfortunately I don't have access to mac or windows right now...

[magland]

@oruebel I pushed a new version of lindi to pypi. While I don't understand what could be causing the problem, there is always a chance that something may have been accidentally fixed. :) So it's worth a try to run your tests again with version 0.3.2.

[oruebel]

Thanks, I'll give it a try

[CodyCBakerPhD]

This appears to no longer be an issue