diff --git a/CHANGELOG.md b/CHANGELOG.md index a481b371..f1b83655 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,11 +1,75 @@ # Changelog -## Next +## 0.7.0 (2018-05-01) -**Breaking changes** +[AZTK is now published on pip!](https://pypi.org/project/aztk/) [Documentation has migrated to readthedocs](aztk.readthedocs.io) -- Moved `docker_repo` under a new `toolkit` key. `docker_repo` is now optional if you want to use the default docker images +This release includes a number of breaking changes. [Please follow the migration for upgrading from 0.6.0.](https://aztk.readthedocs.io/en/v0.7.0/80-migration.html). + +**Breaking Changes** + +- Moved `docker_repo` under a new `toolkit` key. `docker_repo` is now only used for custom Docker images. Use toolkit for supported images. +- Docker images have been refactored and moved to a different Dockerhub repository. The new supported images are not backwards compatible. See [the documentation on configuration files.](https://aztk.readthedocs.io/en/v0.7.0/13-configuration.html#cluster-yaml) + +**Deprecated Features** +- Custom scripts have been removed in favor of Plugins, which are more robust. See, [the documenation on Plugins.](https://aztk.readthedocs.io/en/v0.7.0/15-plugins.html) + +**Added Features** +* add internal flag to node commands (#482) ([1eaa1b6](https://github.com/Azure/aztk/commit/1eaa1b6)), closes [#482](https://github.com/Azure/aztk/issues/482) +* Added custom scripts functionality for plugins with the cli(Deprecate custom scripts) (#517 ([c98df7d](https://github.com/Azure/aztk/commit/c98df7d)), closes [#517](https://github.com/Azure/aztk/issues/517) +* disable msrestazure keyring log (#509) ([3cc43c3](https://github.com/Azure/aztk/commit/3cc43c3)), closes [#509](https://github.com/Azure/aztk/issues/509) +* enable mixed mode for jobs (#442) ([8d00a2c](https://github.com/Azure/aztk/commit/8d00a2c)), closes [#442](https://github.com/Azure/aztk/issues/442) +* getting started script (#475) ([7ef721f](https://github.com/Azure/aztk/commit/7ef721f)), closes [#475](https://github.com/Azure/aztk/issues/475) +* JupyterLab plugin (#459) ([da61337](https://github.com/Azure/aztk/commit/da61337)), closes [#459](https://github.com/Azure/aztk/issues/459) +* managed storage for clusters and jobs (#443) ([8aa1843](https://github.com/Azure/aztk/commit/8aa1843)), closes [#443](https://github.com/Azure/aztk/issues/443) +* match cluster submit exit code in cli (#478) ([8889059](https://github.com/Azure/aztk/commit/8889059)), closes [#478](https://github.com/Azure/aztk/issues/478) +* Plugin V2: Running plugin on host (#461) ([de78983](https://github.com/Azure/aztk/commit/de78983)), closes [#461](https://github.com/Azure/aztk/issues/461) +* Plugins (#387) ([c724d94](https://github.com/Azure/aztk/commit/c724d94)), closes [#387](https://github.com/Azure/aztk/issues/387) +* Pypi auto deployement (#428) ([c237501](https://github.com/Azure/aztk/commit/c237501)), closes [#428](https://github.com/Azure/aztk/issues/428) +* Readthedocs support (#497) ([e361c3b](https://github.com/Azure/aztk/commit/e361c3b)), closes [#497](https://github.com/Azure/aztk/issues/497) +* refactor docker images (#510) ([779bffb](https://github.com/Azure/aztk/commit/779bffb)), closes [#510](https://github.com/Azure/aztk/issues/510) +* Spark add output logs flag (#468) ([32de752](https://github.com/Azure/aztk/commit/32de752)), closes [#468](https://github.com/Azure/aztk/issues/468) +* spark debug tool (#455) ([44a0765](https://github.com/Azure/aztk/commit/44a0765)), closes [#455](https://github.com/Azure/aztk/issues/455) +* spark ui proxy plugin (#467) ([2e995b4](https://github.com/Azure/aztk/commit/2e995b4)), closes [#467](https://github.com/Azure/aztk/issues/467) +* Spark vnet custom dns hostname fix (#490) ([61e7c59](https://github.com/Azure/aztk/commit/61e7c59)), closes [#490](https://github.com/Azure/aztk/issues/490) +* New Toolkit configuration (#507) ([7a7e63c](https://github.com/Azure/aztk/commit/7a7e63c)), closes [#507](https://github.com/Azure/aztk/issues/507) + +**Bug Fixes** +* add gitattributes file (#470) ([82ad029](https://github.com/Azure/aztk/commit/82ad029)), closes [#470](https://github.com/Azure/aztk/issues/470) +* add plugins to cluster_install_cmd call (#423) ([216f63d](https://github.com/Azure/aztk/commit/216f63d)), closes [#423](https://github.com/Azure/aztk/issues/423) +* add spark.history.fs.logDirectory to required keys (#456) ([4ef3dd0](https://github.com/Azure/aztk/commit/4ef3dd0)), closes [#456](https://github.com/Azure/aztk/issues/456) +* add support for jars, pyfiles, files in Jobs (#408) ([2dd7891](https://github.com/Azure/aztk/commit/2dd7891)), closes [#408](https://github.com/Azure/aztk/issues/408) +* add timeout handling to cluster_run and copy (#524) ([47000a5](https://github.com/Azure/aztk/commit/47000a5)), closes [#524](https://github.com/Azure/aztk/issues/524) +* azure file share not being shared with container (#521) ([07ac9b7](https://github.com/Azure/aztk/commit/07ac9b7)), closes [#521](https://github.com/Azure/aztk/issues/521) +* Dependency issue with keyring not having good dependencies (#504) ([5e79a2c](https://github.com/Azure/aztk/commit/5e79a2c)), closes [#504](https://github.com/Azure/aztk/issues/504) +* filter job submission clusters out of cluster list (#409) ([1c31335](https://github.com/Azure/aztk/commit/1c31335)), closes [#409](https://github.com/Azure/aztk/issues/409) +* fix aztk cluster submit paths, imports (#464) ([c1f43c7](https://github.com/Azure/aztk/commit/c1f43c7)), closes [#464](https://github.com/Azure/aztk/issues/464) +* fix broken spark init command (#486) ([a33bdbc](https://github.com/Azure/aztk/commit/a33bdbc)), closes [#486](https://github.com/Azure/aztk/issues/486) +* fix job submission cluster data issues (#533) ([9ccc1c6](https://github.com/Azure/aztk/commit/9ccc1c6)), closes [#533](https://github.com/Azure/aztk/issues/533) +* fix spark job submit path (#474) ([ee1e61b](https://github.com/Azure/aztk/commit/ee1e61b)), closes [#474](https://github.com/Azure/aztk/issues/474) +* make node scripts upload in memory (#519) ([0015e22](https://github.com/Azure/aztk/commit/0015e22)), closes [#519](https://github.com/Azure/aztk/issues/519) +* pypi long description (#450) ([db7a2ef](https://github.com/Azure/aztk/commit/db7a2ef)), closes [#450](https://github.com/Azure/aztk/issues/450) +* remove unnecessary example (#417) ([f1e3f7a](https://github.com/Azure/aztk/commit/f1e3f7a)), closes [#417](https://github.com/Azure/aztk/issues/417) +* Remove unused ssh plugin flags (#488) ([be8cd2a](https://github.com/Azure/aztk/commit/be8cd2a)), closes [#488](https://github.com/Azure/aztk/issues/488) +* set explicit file open encoding (#448) ([5761a36](https://github.com/Azure/aztk/commit/5761a36)), closes [#448](https://github.com/Azure/aztk/issues/448) +* Spark shuffle service worker registration fail (#492) ([013f6e4](https://github.com/Azure/aztk/commit/013f6e4)), closes [#492](https://github.com/Azure/aztk/issues/492) +* throw error if submitting before master elected (#479) ([a59fe8b](https://github.com/Azure/aztk/commit/a59fe8b)), closes [#479](https://github.com/Azure/aztk/issues/479) +* hdfs using wrong conditions (#515) ([a00dbb7](https://github.com/Azure/aztk/commit/a00dbb7)), closes [#515](https://github.com/Azure/aztk/issues/515) +* AZTK_IS_MASTER not set on worker and failing (#506) ([b8a3fcc](https://github.com/Azure/aztk/commit/b8a3fcc)), closes [#506](https://github.com/Azure/aztk/issues/506) +* VNet required error now showing if using mixed mode without it (#440) ([9253aac](https://github.com/Azure/aztk/commit/9253aac)), closes [#440](https://github.com/Azure/aztk/issues/440) +* Worker on master flag ignored and standardize boolean environment (#514) ([5579d95](https://github.com/Azure/aztk/commit/5579d95)), closes [#514](https://github.com/Azure/aztk/issues/514) +* Fix job configuration option for `aztk spark job submit` command (#435) ([4be5ac2](https://github.com/Azure/aztk/commit/4be5ac2)), closes [#435](https://github.com/Azure/aztk/issues/435) +* Fix keyring (#505) ([12450fb](https://github.com/Azure/aztk/commit/12450fb)), closes [#505](https://github.com/Azure/aztk/issues/505) +* Fix the endpoint (#437) ([bcefca3](https://github.com/Azure/aztk/commit/bcefca3)), closes [#437](https://github.com/Azure/aztk/issues/437) +* Fix typo in command_builder 'expecity' -> 'explicitly' (#447) ([27822f4](https://github.com/Azure/aztk/commit/27822f4)), closes [#447](https://github.com/Azure/aztk/issues/447) +* Fix typo load_aztk_screts -> load_aztk_secrets (#421) ([6827181](https://github.com/Azure/aztk/commit/6827181)), closes [#421](https://github.com/Azure/aztk/issues/421) +* Update file to point at master branch (#501) ([4ba3c9d](https://github.com/Azure/aztk/commit/4ba3c9d)), closes [#501](https://github.com/Azure/aztk/issues/501) +* Update storage sdk from 0.33.0 to 1.1.0 (#439) ([f2eb1a4](https://github.com/Azure/aztk/commit/f2eb1a4)), closes [#439](https://github.com/Azure/aztk/issues/439) + +**Internal Changes** +* Internal: Cluster data helpers and upload_node_script into cluster_data module (#401) ([2bed496](https://github.com/Azure/aztk/commit/2bed496)), closes [#401](https://github.com/Azure/aztk/issues/401) +* Internal: Move node scripts under aztk and upload all aztk to cluster (#433) ([dfbfead](https://github.com/Azure/aztk/commit/dfbfead)), closes [#433](https://github.com/Azure/aztk/issues/433) ## 0.6.0 Mixed Mode, Cluster Run & Copy diff --git a/aztk/models/models.py b/aztk/models/models.py index 70d2f123..14de11b9 100644 --- a/aztk/models/models.py +++ b/aztk/models/models.py @@ -152,7 +152,7 @@ def validate(self) -> bool: ) if self.custom_scripts: - deprecate("Custom scripts are DEPRECATED and will be removed in 0.8.0. Use plugins instead See https://aztk.readthedocs.io/en/latest/15-plugins.html") + deprecate("Custom scripts are DEPRECATED and will be removed in 0.8.0. Use plugins instead See https://aztk.readthedocs.io/en/v0.7.0/15-plugins.html") class RemoteLogin: diff --git a/aztk/version.py b/aztk/version.py index 32dd4af6..ec660821 100644 --- a/aztk/version.py +++ b/aztk/version.py @@ -25,7 +25,7 @@ minor = 7 patch = 0 -suffix = 'b20' +suffix = '' __version__ = "{major}.{minor}.{patch}{suffix}".format( major=major, diff --git a/docs/12-docker-image.md b/docs/12-docker-image.md index 20d61a2e..df921ed5 100644 --- a/docs/12-docker-image.md +++ b/docs/12-docker-image.md @@ -90,6 +90,6 @@ ENV PATH $SPARK_HOME/bin:$PATH If you are using your own version of Spark, make that it is symlinked by "/home/spark-current". **$SPARK_HOME**, must also point to "/home/spark-current". ## Hosting your Docker Image -By default, this toolkit assumes that your Docker images are publicly hosted on Docker Hub. However, we also support hosting your images privately. +By default, aztk assumes that your Docker images are publicly hosted on Docker Hub. However, we also support hosting your images privately. See [here](https://github.com/Azure/aztk/blob/v0.7.0/docs/12-docker-image.md#using-a-custom-docker-image-that-is-privately-hosted) to learn more about using privately hosted Docker Images. diff --git a/docs/13-configuration.md b/docs/13-configuration.md index 51fa28d9..294fc9dc 100644 --- a/docs/13-configuration.md +++ b/docs/13-configuration.md @@ -110,8 +110,6 @@ The following settings available in `spark-defaults.conf` and `spark-env.sh` are `spark-defaults.conf`: - spark.master -Also note that this toolkit pre-loads wasb jars, so loading them elsewhere is not necessary. - ### History Server If you want to use Spark's history server, please set the following values in your `.aztk/spark-defaults.conf` file: ``` diff --git a/docs/80-migration.md b/docs/80-migration.md new file mode 100644 index 00000000..a779fa12 --- /dev/null +++ b/docs/80-migration.md @@ -0,0 +1,48 @@ +# Migration Guide + + +## 0.6.0 to 0.7.0 +This guide will describe the steps needed to update a 0.6.0 aztk installation to 0.7.0. + +## Installation from pip +[AZTK is now published on pip!](https://pypi.org/project/aztk/) If you installed from github previously, please reinstall. + +To uninstall run: +``` +pip3 uninstall aztk +``` + +The following command will get the latest release of aztk (please ensure you are using python3.5+): +``` +pip3 install aztk +``` +Or, you can install 0.7.0 specifically using: +``` +pip3 install aztk==0.7.0 +``` + +## Configuration Files +A number of changes have been made that affect previously init'ed aztk environments. To limit potential issues with previous versions, we recommend that you replace any existing `.aztk` directories. + +1. Backup your existing `.aztk` directory by renaming it to `.aztk.old`. +2. Run `aztk spark init` to create a new `.aztk` directory +3. Copy the **values** from `.aztk.old/secrets.yaml` to `.aztk/secrets.yaml` +4. Update the new `.aztk/cluster.yaml` with values from `.aztk.old/cluster.yaml` if applicable. Please be aware of the new `toolkit` section that replaces `docker_repo` for supported images. Similarly for `.aztk/job.yaml`. +5. Update the new defaults in `.aztk/spark-defaults.conf`, `.aztk/core-site.xml` and `.aztk/spark-env.sh` if applicable. +6. Be sure to **not** copy over the `.aztk.old/jars` directory. All jars that were placed here by default have been moved on the Docker image. You can add any custom jars you had by placing them in `.aztk/jars/`. +7. Create your new 0.7.0 cluster! + +### cluster.yaml +In cluster.yaml, the `toolkit` key has been added. It is used to select the default, supported Docker images. Please refer to [the configuration file documentation.](https://aztk.readthedocs.io/en/v0.7.0/13-configuration.html#cluster-yaml) + +## Docker images +A major backwards-incompatible refactor of the Docker images has occurred. Previous Docker images will no longer work with 0.7.0. To update to a new supported docker image, you will need to update your `.aztk/cluster.yaml` configuration file with the `toolkit` block in place of `docker_repo`. If you do not do so, cluster creation will fail! + +Please refer to the [the configuration file documentation](https://aztk.readthedocs.io/en/v0.7.0/13-configuration.html#cluster-yaml) for more information on the `toolkit` in `cluster.yaml`. + + +## Custom scripts depreciation and Plugins +Custom scripts have been depreciated in favor of Plugins. Plugins have a number of advantages, including the ability to execute on the host (and not in the Spark Docker container). A number of supported plugins are shipped with aztk, please refer to [the plugin documentation to learn more.](https://aztk.readthedocs.io/en/v0.7.0/15-plugins.html) + +## Read the Docs +[Documentation has migrated to readthedocs](https://aztk.readthedocs.io/en/v0.7.0). \ No newline at end of file diff --git a/docs/index.rst b/docs/index.rst index 7e82ee2b..68a6d51b 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -22,6 +22,7 @@ This toolkit is built on top of Azure Batch but does not require any Azure Batch 30-cloud-storage 60-gpu 70-jobs + 80-migration .. _sdk-docs: .. toctree::