FEAT-modin-project#4412: Add Batch Pipeline API to Modin (modin-proje…

…ct#4452) Co-authored-by: Yaroslav Igoshev <[email protected]> Co-authored-by: Mahesh Vashishtha <[email protected]> Signed-off-by: Rehan Durrani <[email protected]>
naren-ponder · Jun 1, 2022 · 3d4404e · 3d4404e
1 parent 312c443
commit 3d4404e
Show file tree

Hide file tree

Showing 13 changed files with 1,386 additions and 5 deletions.
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
@@ -119,6 +119,7 @@ jobs:
             modin/experimental/core/execution/native/implementations/omnisci_on_native/omnisci_worker.py \
       - run: python scripts/doc_checker.py modin/experimental/core/storage_formats/omnisci
       - run: python scripts/doc_checker.py modin/experimental/core/execution/native/implementations/omnisci_on_native/exchange/dataframe_protocol
+      - run: python scripts/doc_checker.py modin/experimental/batch/pipeline.py
       - run: python scripts/doc_checker.py modin/logging
 
   lint-flake8:
@@ -462,6 +463,7 @@ jobs:
         if: matrix.engine == 'ray'
       - run: pytest -n 2 modin/experimental/xgboost/test/test_dmatrix.py
         if: matrix.engine == 'ray'
+      - run: pytest -n 2 modin/experimental/batch/test/test_pipeline.py
       - run: pytest -n 2 modin/pandas/test/dataframe/test_binary.py
       - run: pytest -n 2 modin/pandas/test/dataframe/test_default.py
       - run: pytest -n 2 modin/pandas/test/dataframe/test_indexing.py

diff --git a/docs/development/architecture.rst b/docs/development/architecture.rst
@@ -338,7 +338,8 @@ details. The documentation covers most modules, with more docs being added every
    │   │   ├─── :doc:`sklearn </flow/modin/experimental/sklearn>`
    │   │   ├───spreadsheet
    │   │   ├───sql
-   │   │   └─── :doc:`xgboost </flow/modin/experimental/xgboost>`
+   │   │   ├─── :doc:`xgboost </flow/modin/experimental/xgboost>`
+   │   │   └─── :doc:`batch </flow/modin/experimental/batch>`
    │   └───pandas
    │       ├─── :doc:`dataframe </flow/modin/pandas/dataframe>`
    │       └─── :doc:`series </flow/modin/pandas/series>`

diff --git a/docs/flow/modin/experimental/batch.rst b/docs/flow/modin/experimental/batch.rst
@@ -0,0 +1,12 @@
+Batch Pipeline API 
+""""""""""""""""""
+
+This API exposes the ability to pipeline row-parallel batch queries on a Modin DataFrame. Currently,
+this feature is only supported for the ``PandasOnRay`` execution.
+
+API
+'''
+
+.. automodule:: modin.experimental.batch.pipeline
+    :members:
+
diff --git a/docs/flow/modin/experimental/experimental.rst b/docs/flow/modin/experimental/experimental.rst
@@ -7,5 +7,5 @@ In some cases Modin can give the user the opportunity to extend (not modify) typ
 API or to try new functionality in order to get more flexibility. Depending on the exact
 experimental feature user may need to install additional packages, change configurations or
 replace the standard Modin import statement ``import modin.pandas as pd`` with modified version
-``import modin.experimental.pandas as pd``. For concreate experimental feature example, please
+``import modin.experimental.pandas as pd``. For concrete examples of experimental features, please
 refer to the desired link from the :ref:`directory tree <directory-tree>`.
diff --git a/docs/flow/modin/experimental/index.rst b/docs/flow/modin/experimental/index.rst
@@ -8,10 +8,12 @@ and provides a limited set of functionality:
 
 * :doc:`xgboost <xgboost>`
 * :doc:`sklearn <sklearn>`
+* :doc:`batch <batch>`
 
 
 .. toctree::
     :hidden:
 
     sklearn
     xgboost
+    batch
diff --git a/docs/release_notes/release_notes-0.15.0.rst b/docs/release_notes/release_notes-0.15.0.rst
@@ -59,6 +59,8 @@ Key Features and Updates
   * FIX-#4390: Add `redis` to Modin dependencies (#4396)
   * FIX-#3689: Add black and flake8 into development environment files (#4480)
   * TEST-#4516: Add numpydoc to developer requirements (#4517)
+* New Features
+  * FEAT-#4412: Add Batch Pipeline API to Modin (#4452)
 
 Contributors
 ------------
@@ -76,3 +78,4 @@ Contributors
 @jrsacher
 @orcahmlee
 @naren-ponder
+@RehanSD
diff --git a/docs/release_notes/release_notes-template.rst b/docs/release_notes/release_notes-template.rst
@@ -27,6 +27,7 @@ Key Features and Updates
   *
 * Dependencies
   *
+* New Features
 
 Contributors
 ------------