ValueError: cannot reindex or align along dimension 'delta_time' because of conflicting dimension sizes: {1112, 1098}

[eliskasieglova]

Hi!

I am having trouble reading ATL08 data. I followed the tutorial both for downloading and reading the data. This is my code:

short_name = 'ATL08'
spatial_extent = [15, 77, 18, 78]
date_range = ['2020-01-01', '2023-01-01']

# accessing data
region_a = ipx.Query(short_name, spatial_extent, date_range, start_time='00:00:00', end_time='23:59:59')
region_a.avail_granules()
region_a.earthdata_login()
region_a.order_granules(subset=False)
region_a.download_granules('cache/is2_ATL08')

# reading
pattern = "processed_ATL{product:2}_{datetime:%Y%m%d%H%M%S}_{rgt:4}{cycle:2}{orbitsegment:2}_{version:3}_{revision:2}.h5"
reader = ipx.Read(data_source=str(input_path), product=data_product, filename_pattern=pattern)
reader.vars.append(var_list=["h_te_best_fit", "latitude", "longitude"])
ds = reader.load()

This is the error message I get:

C:\Users\eliss\anaconda3\envs\SvalbardSurges\Lib\site-packages\icepyx\core\read.py:49: UserWarning: Converting non-nanosecond precision datetime values to nanosecond precision. This behavior can eventually be relaxed in xarray, as it is an artifact from pandas which is now beginning to support non-nanosecond precision values. This warning is caused by passing non-nanosecond np.datetime64 or np.timedelta64 values to the DataArray or Variable constructor; it can be silenced by converting the values to nanosecond precision ahead of time.
  df.update({keyword: df[keyword].str[:-1].astype(np.datetime64)})
C:\Users\eliss\anaconda3\envs\SvalbardSurges\Lib\site-packages\icepyx\core\read.py:49: UserWarning: Converting non-nanosecond precision datetime values to nanosecond precision. This behavior can eventually be relaxed in xarray, as it is an artifact from pandas which is now beginning to support non-nanosecond precision values. This warning is caused by passing non-nanosecond np.datetime64 or np.timedelta64 values to the DataArray or Variable constructor; it can be silenced by converting the values to nanosecond precision ahead of time.
  df.update({keyword: df[keyword].str[:-1].astype(np.datetime64)})
C:\Users\eliss\anaconda3\envs\SvalbardSurges\Lib\site-packages\icepyx\core\read.py:490: UserWarning: rename 'delta_time' to 'photon_idx' does not create an index anymore. Try using swap_dims instead or use set_index after rename to create an indexed coordinate.
  .rename({"delta_time": "photon_idx"})
C:\Users\eliss\anaconda3\envs\SvalbardSurges\Lib\site-packages\icepyx\core\read.py:490: UserWarning: rename 'delta_time' to 'photon_idx' does not create an index anymore. Try using swap_dims instead or use set_index after rename to create an indexed coordinate.
  .rename({"delta_time": "photon_idx"})
Traceback (most recent call last):
  File "C:\Users\eliss\Documents\SvalbardSurges\main.py", line 205, in <module>
    main()
  File "C:\Users\eliss\Documents\SvalbardSurges\main.py", line 39, in main
    is2_data = is2.read_is2(
               ^^^^^^^^^^^^^
  File "C:\Users\eliss\Documents\SvalbardSurges\svalbardsurges\inputs\is2.py", line 61, in read_is2
    ds = reader.load()
         ^^^^^^^^^^^^^
  File "C:\Users\eliss\anaconda3\envs\SvalbardSurges\Lib\site-packages\icepyx\core\read.py", line 632, in load
    self._build_single_file_dataset(file, groups_list)
  File "C:\Users\eliss\anaconda3\envs\SvalbardSurges\Lib\site-packages\icepyx\core\read.py", line 822, in _build_single_file_dataset
    is2ds = is2ds.merge(ds, join="outer", combine_attrs="no_conflicts")
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\eliss\anaconda3\envs\SvalbardSurges\Lib\site-packages\xarray\core\dataset.py", line 5630, in merge
    merge_result = dataset_merge_method(
                   ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\eliss\anaconda3\envs\SvalbardSurges\Lib\site-packages\xarray\core\merge.py", line 1038, in dataset_merge_method
    return merge_core(
           ^^^^^^^^^^^
  File "C:\Users\eliss\anaconda3\envs\SvalbardSurges\Lib\site-packages\xarray\core\merge.py", line 710, in merge_core
    aligned = deep_align(
              ^^^^^^^^^^^
  File "C:\Users\eliss\anaconda3\envs\SvalbardSurges\Lib\site-packages\xarray\core\alignment.py", line 847, in deep_align
    aligned = align(
              ^^^^^^
  File "C:\Users\eliss\anaconda3\envs\SvalbardSurges\Lib\site-packages\xarray\core\alignment.py", line 783, in align
    aligner.align()
  File "C:\Users\eliss\anaconda3\envs\SvalbardSurges\Lib\site-packages\xarray\core\alignment.py", line 569, in align
    self.assert_unindexed_dim_sizes_equal()
  File "C:\Users\eliss\anaconda3\envs\SvalbardSurges\Lib\site-packages\xarray\core\alignment.py", line 470, in assert_unindexed_dim_sizes_equal
    raise ValueError(
ValueError: cannot reindex or align along dimension 'delta_time' because of conflicting dimension sizes: {1112, 1098}

I am using icepyx 0.7.0., pandas 2.1.1 and xarray 2023.8.0, but I tried it on older versions as well (pandas 1.5.3, xarray 2022.10.0) which did not solve the issue. I also tried running it on a smaller dataset containing only 4 ATL08 files, still get this message.

[JessicaS11]

Hello @eliskasieglova! Thanks for reaching out, and also for providing the full set of code you are running, what you've tried, and some version information. I was able to reproduce this error with an ATL08 file I have locally (NSIDC is doing maintenance today, so I couldn't download any of the files you specified) on our most recent version of icepyx (0.8.0) as well as our current development version. I'll flag this as a bug and hopefully myself or someone on the team can work on it soon! We'd also welcome your collaboration if you're interested in diving into the underlying code.

[JessicaS11]

Hello @eliskasieglova! We just created a new release (v0.8.1) that has a fix for this issue. Please let us know if you still have any issues after upgrading to that version.