Skip to content

Commit

Permalink
Doc: add download link and script in orca installation. (intel#6750)
Browse files Browse the repository at this point in the history
* feat: add download orca in installation.

* fix: fix wording.

* fix: fix typo.

* fix: fix wording.

* fix: fix link and add scripts.

* fix: fix wording.

* fix: fix wording.

* fix: add empty check for env variables.

* fix: fix echo.

* fix: update download link

* fix: fix style.

* fix: fix style.

* fix: fix wording and style.

* fix: fix style.

* fix: fix style.

* fix: fix style.
  • Loading branch information
lalalapotter authored Nov 24, 2022
1 parent 65c4c2a commit 620511e
Show file tree
Hide file tree
Showing 3 changed files with 55 additions and 1 deletion.
12 changes: 12 additions & 0 deletions docs/readthedocs/source/doc/Orca/Overview/install.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,8 @@ conda activate py37
---
## Install BigDL Orca

This section demonstrates how to install BigDL Orca via `pip`, which is the most recommended way.

### To use basic Orca features
You can install Orca in your created conda environment for distributed data processing, training and inference with the following command:
```bash
Expand Down Expand Up @@ -121,3 +123,13 @@ pip uninstall bigdl-orca-spark3 bigdl-dllib-spark3 bigdl-tf bigdl-math bigdl-cor
```

__Note__: If necessary, you need to manually uninstall `pyspark` and other [dependencies](https://github.com/intel-analytics/BigDL/tree/main/python/requirements/orca) introduced by Orca.

---
## Download BigDL Orca

You can also download the BigDL package via the download links below. Besides, you can use the [script](https://github.com/intel-analytics/BigDL/blob/main/scripts/download-bigdl.sh) to download.

| | BigDL 2.1.0 | BigDL 2.2.0-SNAPSHOT nightly |
| :-------: | :---------: | :--------------------------: |
| Spark 2.4 | [download](https://repo1.maven.org/maven2/com/intel/analytics/bigdl/bigdl-assembly-spark_2.4.6/2.1.0/bigdl-assembly-spark_2.4.6-2.1.0-fat-jars.zip) | [download](https://oss.sonatype.org/content/repositories/snapshots/com/intel/analytics/bigdl/bigdl-assembly-spark_2.4.6/2.2.0-SNAPSHOT/) |
| Spark 3.1 | [download](https://repo1.maven.org/maven2/com/intel/analytics/bigdl/bigdl-assembly-spark_3.1.2/2.1.0/bigdl-assembly-spark_3.1.2-2.1.0-fat-jars.zip) | [download](https://oss.sonatype.org/content/repositories/snapshots/com/intel/analytics/bigdl/bigdl-assembly-spark_3.1.3/2.2.0-SNAPSHOT/) |
2 changes: 1 addition & 1 deletion docs/readthedocs/source/doc/Orca/Tutorial/yarn.md
Original file line number Diff line number Diff line change
Expand Up @@ -309,7 +309,7 @@ On the __Client Node__:
export SPARK_HOME=/path/to/spark # the folder path where you extract the Spark package
export SPARK_VERSION="downloaded spark version"
```
2. Download and unzip a BigDL assembly package from [BigDL Assembly Spark 2.4.6](https://repo1.maven.org/maven2/com/intel/analytics/bigdl/bigdl-assembly-spark_2.4.6/2.1.0/bigdl-assembly-spark_2.4.6-2.1.0-fat-jars.zip) or [BigDL Assembly Spark 3.1.2](https://repo1.maven.org/maven2/com/intel/analytics/bigdl/bigdl-assembly-spark_3.1.2/2.1.0/bigdl-assembly-spark_3.1.2-2.1.0-fat-jars.zip) (according to your Spark version), then setup the environment variables `${BIGDL_HOME}` and `${BIGDL_VERSION}`.
2. Refer to [here](../Overview/install.html#download-bigdl-orca) to download and unzip a BigDL assembly package. Make sure the Spark version of your downloaded BigDL matches your downloaded Spark. Then setup the environment variables `${BIGDL_HOME}` and `${BIGDL_VERSION}`.
```bash
export BIGDL_HOME=/path/to/unzipped_BigDL
export BIGDL_VERSION="downloaded BigDL version"
Expand Down
42 changes: 42 additions & 0 deletions scripts/download-bigdl.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
#!/bin/bash

#
# Copyright 2016 The BigDL Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

set -x

if [ -z "$BIGDL_VERSION" ]; then
echo "Please specify BIGDL_VERSION, e.g. 2.1.0, 2.2.0-SNAPSHOT."
exit 1
fi

echo $BIGDL_VERSION

if [ -z "$SPARK_VERSION" ]; then
echo "Please specify SPARK_VERSION, e.g. 2.4.6, 3.1.2 and 3.1.3."
exit 1
fi

echo $SPARK_VERSION

if [[ $BIGDL_VERSION == *"SNAPSHOT"* ]]; then
NIGHTLY_VERSION=$(echo $(echo `wget -qO - https://oss.sonatype.org/content/repositories/snapshots/com/intel/analytics/bigdl/bigdl-assembly-spark_$SPARK_VERSION/$BIGDL_VERSION/maven-metadata.xml | sed -n '/<value>[0-9]*\.[0-9]*\.[0-9]*-[0-9][0-9]*\.[0-9][0-9]*-[0-9][0-9]*.*value>/p' | head -n1 | awk -F'>' '{print $2}' | tr '</value' ' '`))
wget https://oss.sonatype.org/content/repositories/snapshots/com/intel/analytics/bigdl/bigdl-assembly-spark_$SPARK_VERSION/$BIGDL_VERSION/bigdl-assembly-spark_$SPARK_VERSION-$NIGHTLY_VERSION-fat-jars.zip
unzip bigdl-assembly-spark_$SPARK_VERSION-$NIGHTLY_VERSION-fat-jars.zip -d $BIGDL_HOME
else
wget https://repo1.maven.org/maven2/com/intel/analytics/bigdl/bigdl-assembly-spark_$SPARK_VERSION/$BIGDL_VERSION/bigdl-assembly-spark_$SPARK_VERSION-$BIGDL_VERSION-fat-jars.zip
unzip bigdl-assembly-spark_$SPARK_VERSION-$BIGDL_VERSION-fat-jars.zip -d $BIGDL_HOME
fi

0 comments on commit 620511e

Please sign in to comment.