Merge pull request #1125 from vrancurel/deprecate_interactive_api

deprecate interactive API

.github/dependabot.yml

@@ -18,9 +18,3 @@ updates:
       interval: "daily"
     ignore:
       - dependency-name: "*"
-  - package-ecosystem: pip
-    directory: /openfl-tutorials/interactive_api
-    schedule:
-      interval: "daily"
-    ignore:
-      - dependency-name: "*"

.github/workflows/ubuntu.yml

@@ -47,28 +47,8 @@ jobs:
         coverage run -m pytest -rA
         coverage report
   
-  interactive-kvasir: # from interactive-kvasir.yml
-    needs: [lint, pytest-coverage]
-    runs-on: ubuntu-latest
-    steps:
-    - uses: actions/checkout@v3
-    - name: Set up Python 3.8
-      uses: actions/setup-python@v3
-      with:
-        python-version: "3.8"
-    - name: Install dependencies
-      run: |
-        python -m pip install --upgrade pip
-        pip install .
-    - name: Interactive API - pytorch_kvasir_unet
-      run: |
-        python setup.py build_grpc
-        pip install torch==1.13.1
-        pip install torchvision==0.14.1
-        python -m tests.github.interactive_api_director.experiments.pytorch_kvasir_unet.run
-  
   cli:
-    needs: [lint, pytest-coverage, interactive-kvasir]
+    needs: [lint, pytest-coverage]
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v3

.github/workflows/windows.yml

@@ -31,28 +31,8 @@ jobs:
         coverage run -m pytest -rA
         coverage report
   
-  interactive-kvasir: # from interactive-kvasir.yml
-    needs: [pytest-coverage]
-    runs-on: windows-latest
-    steps:
-    - uses: actions/checkout@v3
-    - name: Set up Python 3.8
-      uses: actions/setup-python@v3
-      with:
-        python-version: "3.8"
-    - name: Install dependencies
-      run: |
-        python -m pip install --upgrade pip
-        pip install .
-    - name: Interactive API - pytorch_kvasir_unet
-      run: |
-        python setup.py build_grpc
-        pip install torch==1.13.1
-        pip install torchvision==0.14.1
-        python -m tests.github.interactive_api_director.experiments.pytorch_kvasir_unet.run
-  
   cli: # from taskrunner.yml
-    needs: [pytest-coverage, interactive-kvasir]
+    needs: [pytest-coverage]
     runs-on: windows-latest
     steps:
       - uses: actions/checkout@v3

README.md

@@ -17,7 +17,7 @@
   <img alt="Coverity Scan Build Status"
        src="https://scan.coverity.com/projects/29040/badge.svg"/>
 </a>
-[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/intel/openfl/blob/develop/openfl-tutorials/interactive_api/numpy_linear_regression/workspace/SingleNotebook.ipynb)
+[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/intel/openfl/blob/develop/openfl-tutorials/Federated_Pytorch_MNIST_Tutorial.ipynb)
 
 Open Federated Learning (OpenFL) is a Python 3 framework for Federated Learning. OpenFL is designed to be a _flexible_, _extensible_ and _easily learnable_ tool for data scientists. OpenFL is hosted by The Linux Foundation, aims to be community-driven, and welcomes contributions back to the project. 
 
@@ -37,9 +37,6 @@ For more installation options check out the [online documentation](https://openf
 
 OpenFL enables data scientists to set up a federated learning experiment following one of the workflows:
 
-- [Director-based Workflow](https://openfl.readthedocs.io/en/latest/about/features_index/interactive.html):
-Setup long-lived components to run many experiments in series. Recommended for FL research when many changes to model, dataloader, or hyperparameters are expected
-
 - [Aggregator-based Workflow](https://openfl.readthedocs.io/en/latest/about/features_index/taskrunner.html):
 Define an experiment and distribute it manually. All participants can verify model code and [FL plan](https://openfl.readthedocs.io/en/latest/about/features_index/taskrunner.html#federated-learning-plan-fl-plan-settings) prior to execution. The federation is terminated when the experiment is finished
 

docs/about/features.rst

@@ -23,7 +23,7 @@ Task Runner
 
         features_index/taskrunner
 
-Interactive
+Interactive (Deprecated)
     Setup long-lived components to run many experiments in series. Recommended for FL research when many changes to model, dataloader, or hyperparameters are expected.
     Formerly known as the director-based workflow.
     For more info see :doc:`features_index/interactive`
@@ -83,7 +83,6 @@ FedCurv
 
     Use :py:class:`openfl.utilities.fedcurv.torch.FedCurv` to override train function using :code:`.get_penalty()`, :code:`.on_train_begin()`, and :code:`.on_train_end()` methods.
     In addition, you should override default :code:`AggregationFunction` of the train task with :class:`openfl.interface.aggregation_functions.FedCurvWeightedAverage`.
-    See :code:`PyTorch_Histology_FedCurv` tutorial in :code:`../openfl-tutorials/interactive_api` directory for more details.
 
 .. _federated_evaluation:
 
@@ -110,4 +109,4 @@ Quantitatively audit data privacy in statistical and machine learning algorithms
     :hidden:
 
     features_index/privacy_meter
-
+

docs/about/features_index/interactive.rst

@@ -3,9 +3,9 @@
 
 .. _running_interactive:
 
-================
-Interactive API
-================
+============================
+Interactive API (Deprecated)
+============================
 
 A director-based workflow uses long-lived components in a federation. These components continue to be available to distribute more experiments in the federation.
 
@@ -670,4 +670,4 @@ Assigner with additional validation round:
 
 .. toctree
 ..    overview.how_can_intel_protect_federated_learning
-..    overview.what_is_intel_federated_learning
+..    overview.what_is_intel_federated_learning

docs/about/features_index/taskrunner.rst

@@ -112,7 +112,7 @@ Each task subsection contains the following:
 .. _running_the_federation_manual:
 
 
-.. _interactive_api:
+.. _interactive_api (Deprecated):
 
 
 

docs/about/releases.md

@@ -11,7 +11,7 @@
 - **Workflow API enhancements**: Introducing an experimental [Workspace Export](https://github.com/securefederatedai/openfl/blob/develop/openfl-tutorials/experimental/1001_Workspace_Creation_from_JupyterNotebook.ipynb) feature that can be used to transform a Workflow API-based FL experiment into the TaskRunner API format for running in a distributed deployment. There is also groundwork laid for a future FederatedRuntime implementation for Workflow API, in addition to the currently supported LocalRuntime.  
 - **Federated Evaluation**: Federated evaluation allows for the assessment of ML models in a federated learning system by validating the model's performance locally on decentralized collaborator nodes, and then aggregating these metrics to gauge overall effectiveness, without compromising data privacy and security. FE is now officially supported by OpenFL, including [example tutorials](https://openfl.readthedocs.io/en/latest/about/features_index/fed_eval.html) on how to use this new feature (via TaskRunner API). 
 
-- **Expanded AI Accelerator Support**:  Intel® Data Center GPU Max Series support via the Intel® Extension for PyTorch, including examples for training on datasets such as [MNIST](https://github.com/securefederatedai/openfl/blob/develop/openfl-tutorials/experimental/104_MNIST_XPU.ipynb) (via Workflow API) and [TinyImageNet](https://github.com/securefederatedai/openfl/tree/develop/openfl-tutorials/interactive_api/PyTorch_TinyImageNet_XPU) (via Interactive API) 
+- **Expanded AI Accelerator Support**:  Intel® Data Center GPU Max Series support via the Intel® Extension for PyTorch, including examples for training on datasets such as [MNIST](https://github.com/securefederatedai/openfl/blob/develop/openfl-tutorials/experimental/104_MNIST_XPU.ipynb) (via Workflow API) and [TinyImageNet](https://github.com/securefederatedai/openfl/tree/main/openfl-tutorials/deprecated/interactive_api/PyTorch_TinyImageNet_XPU) (via Interactive API) 
 
 - **Improved straggler collaborator handling**: Improvements and bug fixes to aggregator’s fault-tolerance when collaborators stop responding or drop out of a federation. Introducing a cut-off timer-based policy and enabling other policies to be plugged-in. This capability is particularly relevant for large or geo-distributed federations. 
 
@@ -35,9 +35,9 @@ We are excited to announce the release of OpenFL 1.5.1 - our first since moving
 - **Documentation accessibility improvements**: As part of our [Global Accessibility Awareness Day](https://www.intel.com/content/www/us/en/developer/articles/community/open-fl-project-improve-accessibility-for-devs.html) (GAAD) Pledge, the OpenFL project is making strides towards more accessible documentation. This release includes the integration of [Intel® One Mono](https://www.intel.com/content/www/us/en/company-overview/one-monospace-font.html) font, contrast color improvements, formatting improvements, and [new accessibility focused issues](https://github.com/securefederatedai/openfl/issues?q=is%3Aissue+is%3Aopen+accessibility) to take up in the future. 
 - **[Documentation to federate a Generally Nuanced Deep Learning Framework (GaNDLF) model with OpenFL](https://openfl.readthedocs.io/en/latest/running_the_federation_with_gandlf.html)**
 - **New OpenFL Interactive API Tutorials**:
-    - [Linear regression with SciKit-Learn](https://github.com/securefederatedai/openfl/tree/develop/openfl-tutorials/interactive_api/scikit_learn_linear_regression)
-    - [MedMNIST 2D Classification Using FedProx Optimizer](https://github.com/securefederatedai/openfl/blob/develop/openfl-tutorials/interactive_api/PyTorch_FedProx_MNIST/README.md?plain=1)
-    - [PyTorch Linear Regression Example](https://github.com/securefederatedai/openfl/tree/develop/openfl-tutorials/interactive_api/PyTorch_LinearRegression)
+    - [Linear regression with SciKit-Learn](https://github.com/securefederatedai/openfl/tree/main/openfl-tutorials/deprecated/interactive_api/scikit_learn_linear_regression)
+    - [MedMNIST 2D Classification Using FedProx Optimizer](https://github.com/securefederatedai/openfl/tree/main/openfl-tutorials/deprecaed/interactive_api/PyTorch_FedProx_MNIST/README.md?plain=1)
+    - [PyTorch Linear Regression Example](https://github.com/securefederatedai/openfl/tree/main/openfl-tutorials/deprecated/interactive_api/PyTorch_LinearRegression)
 - **Improvements to workspace export and import**
 - **Many documentation improvements and updates**
 - **Bug fixes**
@@ -54,10 +54,10 @@ We are excited to announce the release of OpenFL 1.5.1 - our first since moving
     * **[Vertical Federated Learning Examples](https://github.com/intel/openfl/tree/develop/openfl-tutorials/experimental/Vertical_FL)** 
     * **[Federated Model Watermarking](https://github.com/intel/openfl/blob/develop/openfl-tutorials/experimental/301_MNIST_Watermarking.ipynb)** using the [WAFFLE](https://arxiv.org/pdf/2008.07298.pdf) method 
     * **[Differential Privacy](https://github.com/intel/openfl/tree/develop/openfl-tutorials/experimental/Global_DP)** – Global differentially private federated learning using Opacus library to achieve a differentially private result w.r.t the inclusion or exclusion of any collaborator in the training process. At each round, a subset of collaborators are selected using a Poisson distribution over all collaborators, the selected collaborators perform local training with periodic clipping of their model delta (with respect to the current global model) to bound their contribution to the average of local model updates. Gaussian noise is then added to the average of these local models at the aggregator. This example is implemented in two different but statistically equivalent ways – the lower level API utilizes RDPAccountant and DPDataloader Opacus objects to perform privacy accounting and collaborator selection respectively, whereas the higher level API uses PrivacyEngine Opacus object for collaborator selection and internally utilizes RDPAccountant for privacy accounting. 
-* **[Habana Accelerator Support](https://github.com/intel/openfl/tree/develop/openfl-tutorials/interactive_api/HPU/PyTorch_TinyImageNet)** 
+* **[Habana Accelerator Support](https://github.com/securefederatedai/openfl/tree/main/openfl-tutorials/deprecated/interactive_api/HPU/PyTorch_TinyImageNet)** 
 * **Official support for Python 3.9 and 3.10** 
 * **[EDEN Compression Pipeline](https://github.com/intel/openfl/blob/develop/openfl/pipelines/eden_pipeline.py)**: Communication-Efficient and Robust Distributed Mean Estimation for Federated Learning ([paper link](https://proceedings.mlr.press/v162/vargaftik22a.html)) 
-* **[FLAX Framework Support](https://github.com/intel/openfl/tree/develop/openfl-tutorials/interactive_api/Flax_CNN_CIFAR)**
+* **[FLAX Framework Support](https://github.com/securefederatedai/openfl/tree/main/openfl-tutorials/deprecated/interactive_api/Flax_CNN_CIFAR)**
 * **Improvements to the resiliency and security of the director / envoy infrastructure**: 
     * Optional notification to plan participants to agree to experiment sent to their infrastructure 
     * Improved resistance to loss of network connectivity and failure at various stages of execution
@@ -109,7 +109,7 @@ The OpenFL v1.2 release contains the following updates:
 
  The OpenFL v1.1 release contains the following updates:
 
-- New [Interactive Python API](https://github.com/intel/openfl/blob/develop/openfl-tutorials/interactive_api_tutorials_(experimental)/Pytorch_Kvasir_UNET_workspace/new_python_api_UNET.ipynb) (experimental)
+- New [Interactive Python API](https://github.com/securefederatedai/openfl/tree/main/openfl-tutorials/deprecated/interactive_api) (experimental)
 - Example FedProx algorithm implementation for PyTorch and Tensorflow
 - `AggregationFunctionInterface` for custom aggregation functions 
 - Adds a [Keras-based NLP Example](https://github.com/intel/openfl/tree/develop/openfl-workspace/keras_nlp)

docs/developer_guide/advanced_topics/overriding_agg_fn.rst

@@ -194,8 +194,8 @@ The following is an example of a **plan.yaml** with a modified aggregation funct
         - loss
 
 
-Interactive API
-================
+Interactive API (Deprecated)
+============================
 You can override aggregation function that will be used for the task this function corresponds to.
 In order to do this, call the ``set_aggregation_function`` decorator method of ``TaskInterface`` and pass ``AggregationFunction`` subclass instance as a parameter.
 For example, you can try:

docs/developer_guide/advanced_topics/overriding_plan_settings.rst

@@ -8,7 +8,7 @@ Updating plan settings
 ***********************
 
 With the director-based workflow, you can use custom plan settings before starting the experiment. Changing plan settings in command line interface is straightforward by modifying plan.yaml.
-When using Python API or Director Envoy based interactive API, **override_config** can be used to update plan settings. 
+When using Python API or Director Envoy based interactive API (Deprecated), **override_config** can be used to update plan settings. 
 
 
 Python API
@@ -24,8 +24,8 @@ Modify the plan settings:
     })
 
 
-Director Envoy Based Interactive API Interface
-==============================================
+Director Envoy Based Interactive API Interface (Deprecated)
+===========================================================
 Once you create an FL_experiment object, a basic federated learning plan with default settings is created. To check the default plan settings, print the plan as shown below:
 
 .. code-block:: python
@@ -96,4 +96,4 @@ Since 'aggregator.settings.db_store_rounds' and 'compression_pipeline.template'
     INFO     Did not find compression_pipeline.settings.n_clusters in config. Make sure it should exist. Creating...                        native.py:105
 
 
-A full implementation can be found at `Federated_Pytorch_MNIST_Tutorial.ipynb <https://github.com/intel/openfl/blob/develop/openfl-tutorials/Federated_Pytorch_MNIST_Tutorial.ipynb>`_ and at `Tensorflow_MNIST.ipynb <https://github.com/intel/openfl/blob/develop/openfl-tutorials/interactive_api/Tensorflow_MNIST/workspace/Tensorflow_MNIST.ipynb>`_.
+A full implementation can be found at `Federated_Pytorch_MNIST_Tutorial.ipynb <https://github.com/intel/openfl/blob/develop/openfl-tutorials/Federated_Pytorch_MNIST_Tutorial.ipynb>`_ and at `Tensorflow_MNIST.ipynb <https://github.com/securefederatedai/openfl/tree/main/openfl-tutorials/deprecated/interactive_api/Tensorflow_MNIST/workspace/Tensorflow_MNIST.ipynb>`_.

docs/developer_guide/structure/components.rst

@@ -49,7 +49,7 @@ The Collaborator is a short-lived entity that manages training the model on loca
     - exchanging model parameters with the Aggregator.
 
 The Collaborator is created by the :ref:`Envoy <openfl_ll_components_envoy>` when a new experiment is submitted
-in the :ref:`Director-based workflow <running_interactive>`. The Collaborator should be started from CLI if a user follows the
+in the :ref:`Director-based workflow <running_interactive>` (Deprecated). The Collaborator should be started from CLI if a user follows the
 :ref:`Aggregator-based workflow <running_the_task_runner>`
 
 Every Collaborator is a unique service. The data loader is loaded with a local *shard descriptor* to perform tasks
@@ -67,7 +67,7 @@ they would like see supported in |productName|.
 Long-Lived Components
 ======================
 
-These components were introduced to support the :ref:`Director-based workflow <running_interactive>`.
+These components were introduced to support the :ref:`Director-based workflow <running_interactive>` (Deprecated).
 
     - The *Director* is the central node of the federation. This component starts an *Aggregator* for each experiment, broadcasts experiment archive to connected collaborator nodes, and provides updates on the status.
     - The *Envoy* runs on collaborator nodes and is always connected to the *Director*. When the *Director* starts an experiment, the *Envoy* starts the *Collaborator* to train the global model.
@@ -81,7 +81,7 @@ Director
 
 The Director is a long-lived entity and is the central node of the federation. It accepts connections from:
 
-    - Frontend clients (data scientists using :ref:`interactive_python_api`)
+    - Frontend clients (data scientists using :ref:`interactive_python_api`) (Deprecated)
     - Envoys, if their Shard Descriptors are complient to the same data interface
 
 The Director supports concurrent frontend connections.
@@ -101,7 +101,7 @@ The Envoy is a long-lived entity that runs on collaborator nodes connected to th
 
 Every Envoy is matched to one `shard descriptor <https://github.com/intel/openfl/blob/develop/openfl/interface/interactive_api/shard_descriptor.py>`_
 in order to run. When the Director starts an experiment, the Envoy accepts the experiment workspace,
-prepares the environment, and starts a Collaborator.
+prepares the environment, and starts a Collaborator. (Note this approach is deprecated)
 
 The envoy is also responsible for sending heartbeat messages to the Director. These messages may also include information
 regarding collaborator machine resource utilization. Refer to :ref:`device monitor plugin <device_monitor_plugin>` for details.