GRIB support

[SebBub]

Dear all,

I just installed xugrid and was hoping to be able to access DWD ICON direct model output which is available as GRIB. I downloaded a sample file from https://opendata.dwd.de/weather/nwp/icon/grib/00/tot_prec/ and installed xugrib latest version from conda-forge as well as cfgrib latest version.

When trying

import xugrid as xu
from pathlib import Path

path = Path(r"C:\Users\Sebastian\Massendaten\ICON\icon_global_icosahedral_single-level_2023090700_001_TOT_PREC.grib2")
file = path / "icon_global_icosahedral_single-level_2023090700_001_TOT_PREC.grib2"

ds = xu.open_dataset(file, engine="cfgrib")

My script seems to run perpetually with no error messages shown. From the xugrid API reference I am not sure if GRIB is at all supported and also I am not sure if the ICON Grib files meet UGRID conventions.

When I terminate the process with CTRL+C I get the following output in the console, not sure if this is of any help:

C:\ProgramData\anaconda3\envs\XUGRID\python.exe C:\Users\Sebastian\Documents\Scripts\Sebastian_Skripte\ICON\xugrid_test.py 
Traceback (most recent call last):
  File "C:\ProgramData\anaconda3\envs\XUGRID\Lib\tempfile.py", line 256, in _mkstemp_inner
    fd = _os.open(file, flags, 0o600)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
PermissionError: [Errno 13] Permission denied: 'C:\\ProgramData\\anaconda3\\envs\\XUGRID\\Lib\\site-packages\\xugrid\\ugrid\\__pycache__\\tmpz4pxnyo3'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\Sebastian\Documents\Scripts\Sebastian_Skripte\ICON\xugrid_test.py", line 1, in <module>
    import xugrid as xu
  File "C:\ProgramData\anaconda3\envs\XUGRID\Lib\site-packages\xugrid\__init__.py", line 4, in <module>
    from xugrid.core.common import (
  File "C:\ProgramData\anaconda3\envs\XUGRID\Lib\site-packages\xugrid\core\common.py", line 6, in <module>
    from xugrid.core.wrap import UgridDataArray, UgridDataset
  File "C:\ProgramData\anaconda3\envs\XUGRID\Lib\site-packages\xugrid\core\wrap.py", line 17, in <module>
    from xugrid.conversion import grid_from_dataset, grid_from_geodataframe
  File "C:\ProgramData\anaconda3\envs\XUGRID\Lib\site-packages\xugrid\conversion.py", line 22, in <module>
    from xugrid.ugrid.connectivity import ragged_index
  File "C:\ProgramData\anaconda3\envs\XUGRID\Lib\site-packages\xugrid\ugrid\connectivity.py", line 566, in <module>
    @nb.njit(cache=True, parallel=True)
     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\ProgramData\anaconda3\envs\XUGRID\Lib\site-packages\numba\core\decorators.py", line 234, in wrapper
    disp.enable_caching()
  File "C:\ProgramData\anaconda3\envs\XUGRID\Lib\site-packages\numba\core\dispatcher.py", line 863, in enable_caching
    self._cache = FunctionCache(self.py_func)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\ProgramData\anaconda3\envs\XUGRID\Lib\site-packages\numba\core\caching.py", line 601, in __init__
    self._impl = self._impl_class(py_func)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\ProgramData\anaconda3\envs\XUGRID\Lib\site-packages\numba\core\caching.py", line 333, in __init__
    locator = cls.from_function(py_func, source_path)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\ProgramData\anaconda3\envs\XUGRID\Lib\site-packages\numba\core\caching.py", line 180, in from_function
    self.ensure_cache_path()
  File "C:\ProgramData\anaconda3\envs\XUGRID\Lib\site-packages\numba\core\caching.py", line 107, in ensure_cache_path
    tempfile.TemporaryFile(dir=path).close()
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\ProgramData\anaconda3\envs\XUGRID\Lib\tempfile.py", line 563, in NamedTemporaryFile
    file = _io.open(dir, mode, buffering=buffering,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\ProgramData\anaconda3\envs\XUGRID\Lib\tempfile.py", line 560, in opener
    fd, name = _mkstemp_inner(dir, prefix, suffix, flags, output_type)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\ProgramData\anaconda3\envs\XUGRID\Lib\tempfile.py", line 262, in _mkstemp_inner
    if (_os.name == 'nt' and _os.path.isdir(dir) and
                             ^^^^^^^^^^^^^^^^^^^
  File "<frozen genericpath>", line 39, in isdir
KeyboardInterrupt

Process finished with exit code -1073741510 (0xC000013A: interrupted by Ctrl+C)

Any help on how to open ICON Grib files in Python would be appreciated.

Many thanks and best

Sebastian

[Huite]

Hi @SebBub,

I'm personally not that familiar with GRIB files. If the variables in the resulting xarray dataset conform nicely to the UGRID conventions, it should just work.

You can try opening the file in an alternative manner via just xarray, mostly as a sanity check:

import xarray as xr
import xugrid as xu

path = Path(r"C:\Users\Sebastian\Massendaten\ICON\icon_global_icosahedral_single-level_2023090700_001_TOT_PREC.grib2")
file = path / "icon_global_icosahedral_single-level_2023090700_001_TOT_PREC.grib2"

ds = xr.open_dataset(file, engine="cfgrib")

Then, in a next step:

uds = xu.UgridDataset(ds)

This is essentially identical to what xugrid does internally.

Anyway, what I expect to find is that xarray will open it just fine, and then the UgridDataset constructor will fail.

From your message above, it looks like your numba is hanging or something. This maybe requires purging you numba install and reinstalling. But for now, to test, you can disable numba as follows:

import os
os.environ["NUMBA_DISABLE_JIT"] = "1"

Put this at the top of your script, then import the other packages, then run your code again.

This will make a select number of functionalities very slow, but it should tell us whether Numba is the problem.