[SPARK-40154][PYTHON][DOCS] Correct storage level in Dataframe.cache …

…docstring ### What changes were proposed in this pull request? Corrects the docstring `DataFrame.cache` to give the correct storage level after it changed with Spark 3.0. It seems that the docstring of `DataFrame.persist` was updated, but `cache` was forgotten. ### Why are the changes needed? The doctoring claims that `cache` uses serialised storage, but it actually uses deserialised storage. I confirmed that this is still the case with Spark 3.5.0 using the example code from the Jira ticket. ### Does this PR introduce _any_ user-facing change? Yes, the docstring changes. ### How was this patch tested? The Github actions workflow succeeded. ### Was this patch authored or co-authored using generative AI tooling? No Closes apache#43229 from paulstaab/SPARK-40154. Authored-by: Paul Staab <[email protected]> Signed-off-by: Sean Owen <[email protected]> (cherry picked from commit 94607dd) Signed-off-by: Sean Owen <[email protected]>
pan3793 · Oct 25, 2023 · ecdb69f · ecdb69f
1 parent c730365
commit ecdb69f
Showing 1 changed file with 2 additions and 2 deletions.
diff --git a/python/pyspark/sql/dataframe.py b/python/pyspark/sql/dataframe.py
@@ -1404,7 +1404,7 @@ def foreachPartition(self, f: Callable[[Iterator[Row]], None]) -> None:
         self.rdd.foreachPartition(f)  # type: ignore[arg-type]
 
     def cache(self) -> "DataFrame":
-        """Persists the :class:`DataFrame` with the default storage level (`MEMORY_AND_DISK`).
+        """Persists the :class:`DataFrame` with the default storage level (`MEMORY_AND_DISK_DESER`).
 
         .. versionadded:: 1.3.0
 
@@ -1413,7 +1413,7 @@ def cache(self) -> "DataFrame":
 
         Notes
         -----
-        The default storage level has changed to `MEMORY_AND_DISK` to match Scala in 2.0.
+        The default storage level has changed to `MEMORY_AND_DISK_DESER` to match Scala in 3.0.
 
         Returns
         -------