RAI Workflow Manager v0.0.13

What's Changed

PRs:

• Allow multiple locations for source import/export by @AndrewNepogoda in #23

1. Deleted CLI arguments:

• --run-mode
• --dev-data-dir
• --output-root

2. CLI Arguments replacement with loader.toml:

CLI arguments --run-mode local, --dev-data-dir, --output-root now replaced with local containers in loader.toml:

Arguments

python main.py --run-mode local --dev-data-dir ./data --output-root ./output ...

loader.toml

[[container]]
name="input"
type="local"
data_path="./data"

[[container]]
name="export"
type="local"
data_path="./output"

3. Renamed properties in loader.toml:

• RAI_PROFILE -> rai_profile
• RAI_PROFILE_PATH -> rai_profile_path
• RAI_SDK_HTTP_RETRIES ->rai_sdk_http_retries

4. Azure container configuration in loader.toml

### Azure blob ###
AZURE_EXPORT_ACCOUNT="account"
AZURE_EXPORT_CONTAINER="dev"
AZURE_EXPORT_DATA_PATH="output"
AZURE_EXPORT_SAS="token"

### Azure IMPORT blob ###
AZURE_IMPORT_ACCOUNT="account"
AZURE_IMPORT_CONTAINER="dev"
AZURE_IMPORT_DATA_PATH="data"
AZURE_IMPORT_SAS="token"

Replaced by:

[[container]]
name="export"
type="azure"
account="account"
container="dev"
data_path="output"
sas="token"

[[container]]
name="input"
type="azure"
account="account"
container="dev"
data_path="data"
sas="token"

5. defaultContainer parameter for ConfigureSources and Export steps in batch config

A new defaultContainer is required for ConfigureSources and Export steps and name a container which will be used as default container for import/export.

6. container parameter for sources and exports

container for sources in ConfigureSources and exports in Export steps allow user specify the name of the container which will be used for the source for import/export. This feature allow user specify multiple datasources (local, azure, snowflake(upcoming), s3(in road map)) as inputs and outputs and mix different types of datasources.

Full Changelog: v0.0.12...v0.0.13