Skip to content
This repository has been archived by the owner on Dec 17, 2024. It is now read-only.

Commit

Permalink
Merge pull request #16 from HongW2019/doc
Browse files Browse the repository at this point in the history
[REMOTE-SHUFFLE-15]Reorg and modify the documents for OAP 1.1.0
  • Loading branch information
zhixingheyi-tian authored Apr 30, 2021
2 parents 8d1d453 + 1427541 commit 23bf824
Show file tree
Hide file tree
Showing 14 changed files with 13,041 additions and 774 deletions.
285 changes: 284 additions & 1 deletion shuffle-hadoop/CHANGELOG.md → CHANGELOG.md

Large diffs are not rendered by default.

1,957 changes: 1,957 additions & 0 deletions LICENSE

Large diffs are not rendered by default.

201 changes: 0 additions & 201 deletions LICENSE.txt

This file was deleted.

18 changes: 8 additions & 10 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
##### \* LEGAL NOTICE: Your use of this software and any required dependent software (the "Software Package") is subject to the terms and conditions of the software license agreements for the Software Package, which may also include notices, disclaimers, or license terms for third party or open source software included in or with the Software Package, and your use indicates your acceptance of all such terms. Please refer to the "TPP.txt" or other similarly-named text file included with the Software Package for additional details.

##### \* Optimized Analytics Package for Spark* Platform is under Apache 2.0 (https://www.apache.org/licenses/LICENSE-2.0).

# Remote Shuffle Plugins

## Online Documentation

You can find the all the PMem Spill documents on the [project web page](https://oap-project.github.io/remote-shuffle/).

## Contents
- [Introduction](#introduction)
- [User Guide](#userguide)
You can find the all the Remote Shuffle documents on the [project web page](https://oap-project.github.io/remote-shuffle/).

## Introduction
Remote Shuffle is a Spark* ShuffleManager plugin, shuffling data through a remote datastore, as opposed to vanilla Spark's local-disks.
Expand All @@ -20,18 +20,16 @@ There are two shuffle plugins in this project.
Different from the above general plugin based on Hadoop Filesystem interface, this plugin bases on DAOS Object API.
Thanks to DAOS Distribution Key and Attribute Key, we can improve performance by constructing shuffle output like
below.
![](./shuffle-daos/images/shuffle.png)
![](./docs/image/shuffle.png)

### Installation
We have provided a Conda package which will automatically install dependencies needed by OAP, you can refer to [OAP-Installation-Guide](./docs/OAP-Installation-Guide.md) for more information. If you have finished [OAP-Installation-Guide](./docs/OAP-Installation-Guide.md), you can find compiled OAP jars in `$HOME/miniconda2/envs/oapenv/oap_jars/`.

## Developer Guide
### Build and Deploy

We have provided a Conda package which will automatically install dependencies needed by OAP, you can refer to [OAP-Installation-Guide](./docs/OAP-Installation-Guide.md) for more information. If you have finished [OAP-Installation-Guide](./docs/OAP-Installation-Guide.md), you can find compiled remote shuffle jars under `$HOME/miniconda2/envs/oapenv/oap_jars`.
Then just skip this section and jump to [User Guide](#user-guide).
Then just skip this section and jump to [User Guide](#g1).

Build using the following command in `OAP/remote-shuffle` folder. This file needs to be deployed on every compute node that runs Spark. Manually place it on all nodes or let resource manager do the work.
Build using the following command in `remote-shuffle` folder. This file needs to be deployed on every compute node that runs Spark. Manually place it on all nodes or let resource manager do the work.

```
mvn -DskipTests clean package
Expand Down
Loading

0 comments on commit 23bf824

Please sign in to comment.