Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

data catalog is not working ... #1119

Open
2 tasks done
roeldegoede opened this issue Feb 20, 2025 · 0 comments
Open
2 tasks done

data catalog is not working ... #1119

roeldegoede opened this issue Feb 20, 2025 · 0 comments
Labels
Bug Something isn't working Needs refinement issue still needs refinement

Comments

@roeldegoede
Copy link
Contributor

HydroMT version checks

  • I have checked that this issue has not already been reported.
  • I have checked that this bug exists on the latest version of HydroMT.

Reproducible Example

mamba create -n hydromt -c conda-forge hydromt
mamba activate hydromt

or just in any environment with pip/mamba

pip/mamba install hydromt

from hydromt.model import Model
mod = Model(data_libs=["artifact_data"])

Current behaviour

**Slightly different errors with pip or mamba install:

MAMBA:**

Traceback (most recent call last):
  File "C:\Users\goede_rl\AppData\Local\miniforge3\envs\hydromt\Lib\site-packages\fsspec\registry.py", line 246, in get_filesystem_class
    register_implementation(protocol, _import_class(bit["class"]))
                                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\goede_rl\AppData\Local\miniforge3\envs\hydromt\Lib\site-packages\fsspec\registry.py", line 281, in _import_class
    mod = importlib.import_module(mod)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\goede_rl\AppData\Local\miniforge3\envs\hydromt\Lib\importlib\__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "C:\Users\goede_rl\AppData\Local\miniforge3\envs\hydromt\Lib\site-packages\fsspec\implementations\http.py", line 9, in <module>
    import aiohttp
ModuleNotFoundError: No module named 'aiohttp'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\goede_rl\AppData\Local\miniforge3\envs\hydromt\Lib\site-packages\hydromt\model\model.py", line 101, in __init__
    self.data_catalog = DataCatalog(data_libs=data_libs, **catalog_keys)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\goede_rl\AppData\Local\miniforge3\envs\hydromt\Lib\site-packages\hydromt\data_catalog\data_catalog.py", line 135, in __init__
    self.from_predefined_catalogs(name_or_path)
  File "C:\Users\goede_rl\AppData\Local\miniforge3\envs\hydromt\Lib\site-packages\hydromt\data_catalog\data_catalog.py", line 597, in from_predefined_catalogs
    catalog_path = self.predefined_catalogs[name].get_catalog_file(version)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\goede_rl\AppData\Local\miniforge3\envs\hydromt\Lib\site-packages\hydromt\data_catalog\predefined_catalog.py", line 186, in get_catalog_file
    version = self.versions[-1]
              ^^^^^^^^^^^^^
  File "C:\Users\goede_rl\AppData\Local\miniforge3\envs\hydromt\Lib\site-packages\hydromt\data_catalog\predefined_catalog.py", line 124, in versions
    self._versions = self._get_versions()
                     ^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\goede_rl\AppData\Local\miniforge3\envs\hydromt\Lib\site-packages\hydromt\data_catalog\predefined_catalog.py", line 138, in _get_versions
    keys = self.registry.keys()
           ^^^^^^^^^^^^^
  File "C:\Users\goede_rl\AppData\Local\miniforge3\envs\hydromt\Lib\site-packages\hydromt\data_catalog\predefined_catalog.py", line 110, in registry
    return self.pooch.registry
           ^^^^^^^^^^
  File "C:\Users\goede_rl\AppData\Local\miniforge3\envs\hydromt\Lib\site-packages\hydromt\data_catalog\predefined_catalog.py", line 117, in pooch
    self._load_registry_file()
  File "C:\Users\goede_rl\AppData\Local\miniforge3\envs\hydromt\Lib\site-packages\hydromt\data_catalog\predefined_catalog.py", line 159, in _load_registry_file
    _copy_to_local(f"{self.base_url}/registry.txt", registry_path)
  File "C:\Users\goede_rl\AppData\Local\miniforge3\envs\hydromt\Lib\site-packages\hydromt\_utils\caching.py", line 44, in _copy_to_local
    fs: AbstractFileSystem = url_to_fs(src)[0]
                             ^^^^^^^^^^^^^^
  File "C:\Users\goede_rl\AppData\Local\miniforge3\envs\hydromt\Lib\site-packages\fsspec\core.py", line 403, in url_to_fs
    chain = _un_chain(url, kwargs)
            ^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\goede_rl\AppData\Local\miniforge3\envs\hydromt\Lib\site-packages\fsspec\core.py", line 351, in _un_chain
    cls = get_filesystem_class(protocol)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\goede_rl\AppData\Local\miniforge3\envs\hydromt\Lib\site-packages\fsspec\registry.py", line 248, in get_filesystem_class
    raise ImportError(bit["err"]) from e
ImportError: HTTPFileSystem requires "requests" and "aiohttp" to be installed

If you would install this "aiohttp" and try again, you get the same error as you get straight away with PIP:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\goede_rl\AppData\Local\miniforge3\Lib\site-packages\hydromt\model\model.py", line 101, in __init__
    self.data_catalog = DataCatalog(data_libs=data_libs, **catalog_keys)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\goede_rl\AppData\Local\miniforge3\Lib\site-packages\hydromt\data_catalog\data_catalog.py", line 135, in __init__
    self.from_predefined_catalogs(name_or_path)
  File "C:\Users\goede_rl\AppData\Local\miniforge3\Lib\site-packages\hydromt\data_catalog\data_catalog.py", line 597, in from_predefined_catalogs
    catalog_path = self.predefined_catalogs[name].get_catalog_file(version)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\goede_rl\AppData\Local\miniforge3\Lib\site-packages\hydromt\data_catalog\predefined_catalog.py", line 186, in get_catalog_file
    version = self.versions[-1]
              ^^^^^^^^^^^^^
  File "C:\Users\goede_rl\AppData\Local\miniforge3\Lib\site-packages\hydromt\data_catalog\predefined_catalog.py", line 124, in versions
    self._versions = self._get_versions()
                     ^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\goede_rl\AppData\Local\miniforge3\Lib\site-packages\hydromt\data_catalog\predefined_catalog.py", line 138, in _get_versions
    keys = self.registry.keys()
           ^^^^^^^^^^^^^
  File "C:\Users\goede_rl\AppData\Local\miniforge3\Lib\site-packages\hydromt\data_catalog\predefined_catalog.py", line 110, in registry
    return self.pooch.registry
           ^^^^^^^^^^
  File "C:\Users\goede_rl\AppData\Local\miniforge3\Lib\site-packages\hydromt\data_catalog\predefined_catalog.py", line 117, in pooch
    self._load_registry_file()
  File "C:\Users\goede_rl\AppData\Local\miniforge3\Lib\site-packages\hydromt\data_catalog\predefined_catalog.py", line 159, in _load_registry_file
    _copy_to_local(f"{self.base_url}/registry.txt", registry_path)
  File "C:\Users\goede_rl\AppData\Local\miniforge3\Lib\site-packages\hydromt\_utils\caching.py", line 48, in _copy_to_local
    fs.get(src, str(dst), block_size=block_size)
  File "C:\Users\goede_rl\AppData\Local\miniforge3\Lib\site-packages\fsspec\asyn.py", line 118, in wrapper
    return sync(self.loop, func, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\goede_rl\AppData\Local\miniforge3\Lib\site-packages\fsspec\asyn.py", line 103, in sync
    raise return_result
  File "C:\Users\goede_rl\AppData\Local\miniforge3\Lib\site-packages\fsspec\asyn.py", line 56, in _runner
    result[0] = await coro
                ^^^^^^^^^^
  File "C:\Users\goede_rl\AppData\Local\miniforge3\Lib\site-packages\fsspec\asyn.py", line 669, in _get
    return await _run_coros_in_chunks(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\goede_rl\AppData\Local\miniforge3\Lib\site-packages\fsspec\asyn.py", line 268, in _run_coros_in_chunks
    result, k = await done.pop()
                ^^^^^^^^^^^^^^^^
  File "C:\Users\goede_rl\AppData\Local\miniforge3\Lib\site-packages\fsspec\asyn.py", line 245, in _run_coro
    return await asyncio.wait_for(coro, timeout=timeout), i
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\goede_rl\AppData\Local\miniforge3\Lib\asyncio\tasks.py", line 520, in wait_for
    return await fut
           ^^^^^^^^^
  File "C:\Users\goede_rl\AppData\Local\miniforge3\Lib\site-packages\fsspec\callbacks.py", line 81, in func
    return await fn(path1, path2, callback=child, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\goede_rl\AppData\Local\miniforge3\Lib\site-packages\fsspec\implementations\http.py", line 246, in _get_file
    async with session.get(self.encode_url(rpath), **kw) as r:
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\goede_rl\AppData\Roaming\Python\Python312\site-packages\aiohttp\client.py", line 1202, in get
    self._request(
TypeError: ClientSession._request() got an unexpected keyword argument 'block_size'
>>>

Desired behaviour

It should just initiliaze the data catalog and all its underlying functionalities. Strangely enough, everything seems to work when I install current main branch of hydromt as an editable pakacge with checking it first and then "pip install -e ." However, without being editable, it still gives the same errors.

Additional context

No response

@roeldegoede roeldegoede added Bug Something isn't working Needs refinement issue still needs refinement labels Feb 20, 2025
roeldegoede added a commit to roeldegoede/hydromt that referenced this issue Feb 20, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug Something isn't working Needs refinement issue still needs refinement
Projects
None yet
Development

No branches or pull requests

1 participant