0.6.0

0.6.0 Mixed Mode, Cluster Run & Copy

Features:

• aztk spark init customization flags
• aztk spark cluster run command added
• aztk spark cluster copy command added
• enable Spark dynamic allocation by default
• add SDK support for file-like objects
• add Spark integration tests
• add flag worker_on_master option for cluster and job submission mode
• Spark driver runs on master node for single application job submission mode

Bug Fixes:

• load jars in .aztk/jars/ in job submission mode
• replace outdated error in cluster_create
• fix type error crash if not jars are specificed in job submission
• stop using mutable default parameters
• print job application code if exit_code is 0
• job submission crash if executor or driver cores specified
• wrong error thrown if user added before master node picked

0.5.1

Major Features:

• Job Submission
• AAD Support, see Getting Started
• VNET Support

Breaking changes:

• SecretsConfiguration inputs changed. Check in SDK for the new format

0.5.0

SDK

0.4.1

hotfixes:

• spark submit command bug
• ssh command for linux/macos

0.4.0 (public release)

Deploy and submit jobs against a Spark on Docker cluster that runs on Azure Batch

• container support
• custom script on master / worker / all-nodes
• configuration files
• renamed to aztk / Azure Distributed Data Engineering Toolkit