Client session for fetching data from CATMAID throws an error

[Marei33]

Hi,

I try to fetch data from CATMAID, more precisely I try to download the Seymour CNS EM volume, but I get error messages regarding setting up the client session via aiohttp...

Here is the python code I use:

And that's the error message I get:

What do I have to do to set up the client session correctly?

[schlegelp]

I'm hoping @clbarnes can chime in here as he implemented the current stack download system.

In the meantime: could you please post your code snippet (minus the credentials of course) as a code block by surrounding it with three tick-marks (`)? Saves us from having to copy your code from the screenshot.

[clbarnes]

Error message as text would be great too!

What version of fsspec do you have installed? The error message is pointing to that library, but the usage seems to match with those in the docs of their latest version.

[Marei33]

Yes, sure!

Here is the code:

import xarray as xr
import requests
import aiohttp
import pymaid
from pymaid.client import CatmaidInstance
from pymaid.stack import Stack

# Initialize the CATMAID instance
catmaid_instance = CatmaidInstance(server='https://neurophyla.mrc-lmb.cam.ac.uk/catmaid/drosophila/l1/seymour/', http_user='', http_password='', api_token='')

# Select the stack
stack = Stack.from_catmaid(stack=7, mirror=8, remote_instance=catmaid_instance) # stack=Seymour (isotropic scale), mirror=LMB Cambridge UK (N5)

# select the mirror session because no default value is given
stack.set_mirror_session(mirror=8, session=aiohttp.ClientSession(auth=aiohttp.BasicAuth("", "")))

print(stack.stack_info)

# Fetch the data at the desired scale level (0 for full resolution)
scale_level = 10
print("Download data ...")
data_array = stack.get_scale(scale_level).as_numpy()

print(data_array.shape)

and the error message:

python catmaid_download.py 
INFO  : Global CATMAID instance set. Caching is ON. (pymaid)
/Users/marei/igor-the-larva/seymour-the-cns/catmaid_download.py:16: DeprecationWarning: The object should be created within an async function
  stack.set_mirror_session(mirror=8, session=aiohttp.ClientSession(auth=aiohttp.BasicAuth("", "")))
StackInfo(sid=7, pid=1, ptitle='L1 CNS', stitle='Seymour (isotropic scale)', downsample_factors=[{'x': 1, 'y': 1, 'z': 1}, {'x': 2, 'y': 2, 'z': 1}, {'x': 4, 'y': 4, 'z': 1}, {'x': 16, 'y': 16, 'z': 1}, {'x': 32, 'y': 32, 'z': 2}, {'x': 64, 'y': 64, 'z': 4}, {'x': 128, 'y': 128, 'z': 8}, {'x': 256, 'y': 256, 'z': 16}, {'x': 512, 'y': 512, 'z': 32}, {'x': 1024, 'y': 1024, 'z': 64}, {'x': 2048, 'y': 2048, 'z': 128}, {'x': 4096, 'y': 4096, 'z': 256}, {'x': 8192, 'y': 8192, 'z': 512}], num_zoom_levels=12, translation={'x': 0.0, 'y': 0.0, 'z': 6050.0}, resolution={'x': 3.8, 'y': 3.8, 'z': 50.0}, dimension={'x': 28128, 'y': 31840, 'z': 4841}, comment='Seymour stack where Z is also downscaled once voxels are somewhere near isotropic', description='Seymour stack where Z is also downscaled once voxels are somewhere near isotropic', metadata=None, broken_slices={}, mirrors=[MirrorInfo(id=8, title='LMB Cambridge UK (N5)', image_base='https://neurophyla.mrc-lmb.cam.ac.uk/tiles/0111-8/seymour.n5/volumes/raw/c0/%SCALE_DATASET%/0_1_2', tile_width=256, tile_height=256, tile_source_type=11, file_extension='', position=0)], orientation=<Orientation.XY: 0>, attribution='', canary_location={'x': 0, 'y': 0, 'z': 0}, placeholder_color=Color(r=0.0, g=0.0, b=0.0, a=1.0))
Download data ...
Traceback (most recent call last):
  File "/Users/marei/igor-the-larva/seymour-the-cns/catmaid_download.py", line 24, in <module>
    data_array = stack.get_scale(scale_level).as_numpy()
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/marei/pymaid/pymaid/stack.py", line 517, in get_scale
    return self._get_n5(mirror_info, scale_level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/marei/pymaid/pymaid/stack.py", line 555, in _get_n5
    container = zarr.open(store, "r")
                ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/marei/mambaforge/envs/igor/lib/python3.11/site-packages/zarr/convenience.py", line 127, in open
    if contains_array(_store, path):
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/marei/mambaforge/envs/igor/lib/python3.11/site-packages/zarr/storage.py", line 120, in contains_array
    return key in store
           ^^^^^^^^^^^^
  File "/Users/marei/mambaforge/envs/igor/lib/python3.11/site-packages/zarr/n5.py", line 458, in __contains__
    return "dimensions" in self._load_n5_attrs(key_new)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/marei/mambaforge/envs/igor/lib/python3.11/site-packages/zarr/n5.py", line 517, in _load_n5_attrs
    s = super().__getitem__(path)
        ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/marei/mambaforge/envs/igor/lib/python3.11/site-packages/zarr/storage.py", line 1441, in __getitem__
    return self.map[key]
           ~~~~~~~~^^^^^
  File "/Users/marei/mambaforge/envs/igor/lib/python3.11/site-packages/fsspec/mapping.py", line 155, in __getitem__
    result = self.fs.cat(k)
             ^^^^^^^^^^^^^^
  File "/Users/marei/mambaforge/envs/igor/lib/python3.11/site-packages/fsspec/asyn.py", line 118, in wrapper
    return sync(self.loop, func, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/marei/mambaforge/envs/igor/lib/python3.11/site-packages/fsspec/asyn.py", line 103, in sync
    raise return_result
  File "/Users/marei/mambaforge/envs/igor/lib/python3.11/site-packages/fsspec/asyn.py", line 56, in _runner
    result[0] = await coro
                ^^^^^^^^^^
  File "/Users/marei/mambaforge/envs/igor/lib/python3.11/site-packages/fsspec/asyn.py", line 461, in _cat
    raise ex
  File "/Users/marei/mambaforge/envs/igor/lib/python3.11/site-packages/fsspec/asyn.py", line 245, in _run_coro
    return await asyncio.wait_for(coro, timeout=timeout), i
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/marei/mambaforge/envs/igor/lib/python3.11/asyncio/tasks.py", line 452, in wait_for
    return await fut
           ^^^^^^^^^
  File "/Users/marei/mambaforge/envs/igor/lib/python3.11/site-packages/fsspec/implementations/http.py", line 233, in _cat_file
    session = await self.set_session()
              ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/marei/mambaforge/envs/igor/lib/python3.11/site-packages/fsspec/implementations/http.py", line 135, in set_session
    self._session = await self.get_client(loop=self.loop, **self.client_kwargs)
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: Stack._get_n5.<locals>.<lambda>() got an unexpected keyword argument 'loop'
Exception ignored in: <function ClientSession.__del__ at 0x1386fa8e0>
Traceback (most recent call last):
  File "/Users/marei/mambaforge/envs/igor/lib/python3.11/site-packages/aiohttp/client.py", line 367, in __del__
  File "/Users/marei/mambaforge/envs/igor/lib/python3.11/asyncio/base_events.py", line 1818, in call_exception_handler
AttributeError: 'NoneType' object has no attribute 'error'

@clbarnes The fsspec version I have installed is 2024.3.1

[Marei33]

Hi @clbarnes, have you had a chance to take a look here so far?

[clbarnes]

Sorry - harder to find the time for it now I'm out of research! I tried to take a look tonight but seems like there are network issues somewhere between me and PyPI, been trying in vain just to install dependencies for the last half hour and keep getting timeouts. My seymour credentials have also been disabled, but I can try virtualflybrain's mirror. I'll give it another go when I can.

[clbarnes]

Got things installed eventually but still having network and other issues. However, as VFB doesn't have any N5 stacks there isn't much I can do to debug this, sorry.

[constantinpape]

Hi @clbarnes,
thanks for having a look at this! I dug into the code a bit deeper now, and it looks like the whole integration of aiohttp is not working anymore, since it needs to be handled within an async function. I am not sure how this would fit into the current code base, and it seems to me like this would need changes on the zarr level.

So getting this to run again would likely be a major effort. We will use a work around for now (downloading the data at full-scale and downscaling it on the fly). But it would maybe make sense to mark the n5 functionality as deprecated / not working. cc @schlegelp.

[clbarnes]

That's very frustrating. Is it a library change or a python one? If library, the version could be pinned. It may only be the HTTP auth which is broken (because otherwise you don't have to construct the aiohttp client yourself), although its possible the construction of a client lower down the stack still causes problems.

[clbarnes]

Otherwise, the only solution is clearly to rewrite pymaid to be async-first ;)