[BUG] sort_values on categorical column fails in dask_cudf #11795
Comments
eriknw
added
Needs Triage
shwina
added
Python
|
This works now, but with the caveat that categorical column must be explicitly marked as ordered @rjzamora any idea why dask_cudf behaves differently from cudf w.r.t. the ordering? |
|
Good catch @vyasr - The dask behavior was actually "fixed" recently in dask-expr (dask/dask-expr#1058), but I just realized that the Even with dask-expr fixed, however, your snippet will not work for dask_cudf, because there seems to be a bug in cudf: import cudf as lib # Works for pandas, but not for cudf
df = lib.DataFrame({"a": list("caba"), "b": list(range(4))})
df["a"] = df["a"].astype("category")
df = df.iloc[:2]
df["a"].cat.as_ordered()EDIT: I submitted #15778 to track this. |
|
Update: Latest version of |
|
The chain has gotten a bit long here, let me summarize to make sure I have everything right. #15780 will fix #15778. Once that is merged, will this issue also be fixed in the dask-expr case, or is there still work to be done to generalize dask-expr to work correctly for cudf because dask/dask-expr#1058 wasn't complete? And in either case, do we still expect this to fail for users of the legacy dask API (which I guess isn't too important if we're going to be forced to migrate to dask-expr anyway)? |
|
Summary:
I certainly want to fix categorical sorting for 24.06 if possible, but my current expectation is that we will need to raise an error and tell the user to disable query planning. If I can find a work-around in the next day or so, then we can remove the error. Otherwise, the proper/upstream fix will only apply to 24.08. |
Follow up to #15788 Adds a temporary workaround for sorting on categorical columns in 24.06: We convert only the partitioning column to pandas to calculate divisions. This is related to #11795, but I don't want to "close" that issue until `RepartitionQuantiles` works with cudf-backed data. Authors: - Richard (Rick) Zamora (https://github.com/rjzamora) Approvers: - Charles Blackmon-Luca (https://github.com/charlesbluca) URL: #15801
Describe the bug
ddf.sort_values(col)does not work with adask_cudfDataFrame whencolis categorical.Steps/Code to reproduce bug
Traceback
--------------------------------------------------------------------------- ValueError Traceback (most recent call last) Cell In [7], line 1 ----> 1 ddf.sort_values("a") File ~/miniconda3/envs/cugraph_dev15/lib/python3.9/contextlib.py:79, in ContextDecorator.__call__.<locals>.inner(*args, **kwds) 76 @wraps(func) 77 def inner(*args, **kwds): 78 with self._recreate_cm(): ---> 79 return func(*args, **kwds) File ~/miniconda3/envs/cugraph_dev15/lib/python3.9/site-packages/dask_cudf/core.py:256, in DataFrame.sort_values(self, by, ignore_index, max_branch, divisions, set_divisions, ascending, na_position, sort_function, sort_function_kwargs, **kwargs) 251 if kwargs: 252 raise ValueError( 253 f"Unsupported input arguments passed : {list(kwargs.keys())}" 254 ) --> 256 df = sorting.sort_values( 257 self, 258 by, 259 max_branch=max_branch, 260 divisions=divisions, 261 set_divisions=set_divisions, 262 ignore_index=ignore_index, 263 ascending=ascending, 264 na_position=na_position, 265 sort_function=sort_function, 266 sort_function_kwargs=sort_function_kwargs, 267 ) 269 if ignore_index: 270 return df.reset_index(drop=True) File ~/miniconda3/envs/cugraph_dev15/lib/python3.9/contextlib.py:79, in ContextDecorator.__call__.<locals>.inner(*args, **kwds) 76 @wraps(func) 77 def inner(*args, **kwds): 78 with self._recreate_cm(): ---> 79 return func(*args, **kwds) File ~/miniconda3/envs/cugraph_dev15/lib/python3.9/site-packages/dask_cudf/sorting.py:266, in sort_values(df, by, max_branch, divisions, set_divisions, ignore_index, ascending, na_position, sort_function, sort_function_kwargs) 264 # Step 1 - Calculate new divisions (if necessary) 265 if divisions is None: --> 266 divisions = quantile_divisions(df, by, npartitions) 268 # Step 2 - Perform repartitioning shuffle 269 meta = df._meta._constructor_sliced([0]) File ~/miniconda3/envs/cugraph_dev15/lib/python3.9/contextlib.py:79, in ContextDecorator.__call__.<locals>.inner(*args, **kwds) 76 @wraps(func) 77 def inner(*args, **kwds): 78 with self._recreate_cm(): ---> 79 return func(*args, **kwds) File ~/miniconda3/envs/cugraph_dev15/lib/python3.9/site-packages/dask_cudf/sorting.py:213, in quantile_divisions(df, by, npartitions) 211 dtype = df[col].dtype 212 if dtype != "object": --> 213 divisions[col] = divisions[col].astype("int64") 214 divisions[col].iloc[-1] += 1 215 divisions[col] = divisions[col].astype(dtype) File ~/miniconda3/envs/cugraph_dev15/lib/python3.9/contextlib.py:79, in ContextDecorator.__call__.<locals>.inner(*args, **kwds) 76 @wraps(func) 77 def inner(*args, **kwds): 78 with self._recreate_cm(): ---> 79 return func(*args, **kwds) File ~/miniconda3/envs/cugraph_dev15/lib/python3.9/site-packages/cudf/core/series.py:1857, in Series.astype(self, dtype, copy, errors, **kwargs) 1855 else: 1856 dtype = {self.name: dtype} -> 1857 return super().astype(dtype, copy, errors, **kwargs) File ~/miniconda3/envs/cugraph_dev15/lib/python3.9/site-packages/cudf/core/indexed_frame.py:3264, in IndexedFrame.astype(self, dtype, copy, errors, **kwargs) 3262 except Exception as e: 3263 if errors == "raise": -> 3264 raise e 3265 return self 3267 return self._from_data(data, index=self._index) File ~/miniconda3/envs/cugraph_dev15/lib/python3.9/site-packages/cudf/core/indexed_frame.py:3261, in IndexedFrame.astype(self, dtype, copy, errors, **kwargs) 3258 raise ValueError("invalid error value specified") 3260 try: -> 3261 data = super().astype(dtype, copy, **kwargs) 3262 except Exception as e: 3263 if errors == "raise": File ~/miniconda3/envs/cugraph_dev15/lib/python3.9/contextlib.py:79, in ContextDecorator.__call__.<locals>.inner(*args, **kwds) 76 @wraps(func) 77 def inner(*args, **kwds): 78 with self._recreate_cm(): ---> 79 return func(*args, **kwds) File ~/miniconda3/envs/cugraph_dev15/lib/python3.9/site-packages/cudf/core/frame.py:328, in Frame.astype(self, dtype, copy, **kwargs) 326 dt = dtype.get(col_name, col.dtype) 327 if not is_dtype_equal(dt, col.dtype): --> 328 result[col_name] = col.astype(dt, copy=copy, **kwargs) 329 else: 330 result[col_name] = col.copy() if copy else col File ~/miniconda3/envs/cugraph_dev15/lib/python3.9/site-packages/cudf/core/column/column.py:857, in ColumnBase.astype(self, dtype, **kwargs) 851 dtype = ( 852 pandas_dtypes_alias_to_cudf_alias.get(dtype, dtype) 853 if isinstance(dtype, str) 854 else pandas_dtypes_to_np_dtypes.get(dtype, dtype) 855 ) 856 if _is_non_decimal_numeric_dtype(dtype): --> 857 return self.as_numerical_column(dtype, **kwargs) 858 elif is_categorical_dtype(dtype): 859 return self.as_categorical_column(dtype, **kwargs) File ~/miniconda3/envs/cugraph_dev15/lib/python3.9/site-packages/cudf/core/column/categorical.py:1236, in CategoricalColumn.as_numerical_column(self, dtype, **kwargs) 1235 def as_numerical_column(self, dtype: Dtype, **kwargs) -> NumericalColumn: -> 1236 return self._get_decategorized_column().as_numerical_column(dtype) File ~/miniconda3/envs/cugraph_dev15/lib/python3.9/site-packages/cudf/core/column/string.py:5312, in StringColumn.as_numerical_column(self, dtype, **kwargs) 5310 if out_dtype.kind in {"i", "u"}: 5311 if not libstrings.is_integer(string_col).all(): -> 5312 raise ValueError( 5313 "Could not convert strings to integer " 5314 "type due to presence of non-integer values." 5315 ) 5316 elif out_dtype.kind == "f": 5317 if not libstrings.is_float(string_col).all(): ValueError: Could not convert strings to integer type due to presence of non-integer values.Expected behavior
I expect it to work--that is, match the result of cudf and dask.dataframe.
Environment overview (please complete the following information)
Environment details
Click here to see environment details
Additional context
Encountered in ProperterGraph in cugraph.
The text was updated successfully, but these errors were encountered: