Lab7 Pipelines.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lab 7: Azure ML Pipelines\n",
    "<img src='https://github.com/retkowsky/images/blob/master/AzureMLservicebanniere.png?raw=true'>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Using Azure Machine Learning Pipelines for Batch Inference\n",
    "\n",
    "In this notebook, we will demonstrate how to make predictions on large quantities of data asynchronously using the ML pipelines with Azure Machine Learning. Batch inference (or batch scoring) provides cost-effective inference, with unparalleled throughput for asynchronous applications. Batch prediction pipelines can scale to perform inference on terabytes of production data. Batch prediction is optimized for high throughput, fire-and-forget predictions for a large collection of data.\n",
    "\n",
    "> **Tip**\n",
    "If your system requires low-latency processing (to process a single document or small set of documents quickly), use [real-time scoring](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-consume-web-service) instead of batch prediction.\n",
    "\n",
    "In this example will be take a digit identification model already-trained on MNIST dataset using the [AzureML training with deep learning example notebook](https://github.com/Azure/MachineLearningNotebooks/blob/master/how-to-use-azureml/training-with-deep-learning/train-hyperparameter-tune-deploy-with-keras/train-hyperparameter-tune-deploy-with-keras.ipynb), and run that trained model on some of the MNIST test images in batch.  \n",
    "\n",
    "The input dataset used for this notebook differs from a standard MNIST dataset in that it has been converted to PNG images to demonstrate use of files as inputs to Batch Inference. A sample of PNG-converted images of the MNIST dataset were take from [this repository](https://github.com/myleott/mnist_png). \n",
    "\n",
    "The outline of this notebook is as follows:\n",
    "\n",
    "- Create a DataStore referencing MNIST images stored in a blob container.\n",
    "- Register the pretrained MNIST model into the model registry. \n",
    "- Use the registered model to do batch inference on the images in the data blob container.\n",
    "\n",
    "## Prerequisites\n",
    "If you are using an Azure Machine Learning Notebook VM, you are all set. Otherwise, make sure you go through the configuration Notebook located at https://github.com/Azure/MachineLearningNotebooks first. This sets you up with a working config file that has information on your workspace, subscription id, etc. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Connect to workspace\n",
    "Create a workspace object from the existing workspace. Workspace.from_config() reads the file config.json and loads the details into an object named ws."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Date: 2021-08-27 08:03:37.339718\n"
     ]
    }
   ],
   "source": [
    "import datetime\n",
    "now = datetime.datetime.now()\n",
    "print('Date:', now)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "You are using Azure ML 1.33.0\n"
     ]
    }
   ],
   "source": [
    "import azureml.core\n",
    "print(\"You are using Azure ML\", azureml.core.VERSION)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Workspace name: azuremlworkshop\n",
      "Azure region: eastus\n",
      "Resource group: azuremlworkshop-rg\n"
     ]
    }
   ],
   "source": [
    "from azureml.core import Workspace\n",
    "\n",
    "ws = Workspace.from_config()\n",
    "print('Workspace name: ' + ws.name, \n",
    "      'Azure region: ' + ws.location, \n",
    "      'Resource group: ' + ws.resource_group, sep = '\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create or Attach existing compute resource\n",
    "By using Azure Machine Learning Compute, a managed service, data scientists can train machine learning models on clusters of Azure virtual machines. Examples include VMs with GPU support. In this tutorial, you create Azure Machine Learning Compute as your training environment. The code below creates the compute clusters for you if they don't already exist in your workspace.\n",
    "\n",
    "**Creation of compute takes approximately 5 minutes. If the AmlCompute with that name is already in your workspace the code will skip the creation process.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "creating a new compute target...\n",
      "InProgress......\n",
      "SucceededProvisioning operation finished, operation \"Succeeded\"\n",
      "Succeeded\n",
      "AmlCompute wait for completion finished\n",
      "\n",
      "Minimum number of nodes requested have been provisioned\n",
      "{'currentNodeCount': 0, 'targetNodeCount': 0, 'nodeStateCounts': {'preparingNodeCount': 0, 'runningNodeCount': 0, 'idleNodeCount': 0, 'unusableNodeCount': 0, 'leavingNodeCount': 0, 'preemptedNodeCount': 0}, 'allocationState': 'Resizing', 'allocationStateTransitionTime': '2021-08-27T08:03:39.609000+00:00', 'errors': None, 'creationTime': '2021-08-27T08:03:39.224377+00:00', 'modifiedTime': '2021-08-27T08:04:04.794891+00:00', 'provisioningState': 'Succeeded', 'provisioningStateTransitionTime': None, 'scaleSettings': {'minNodeCount': 0, 'maxNodeCount': 4, 'nodeIdleTimeBeforeScaleDown': 'PT1800S'}, 'vmPriority': 'Dedicated', 'vmSize': 'STANDARD_D2_V2'}\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "from azureml.core.compute import AmlCompute, ComputeTarget\n",
    "\n",
    "# choose a name for your cluster\n",
    "import random\n",
    "randomval=round(random.random()*1000000)\n",
    "compute_name=\"pipelines\"+str(randomval)\n",
    "compute_name\n",
    "\n",
    "# This example uses CPU VM. For using GPU VM, set SKU to STANDARD_NC6\n",
    "vm_size = os.environ.get(\"AML_COMPUTE_CLUSTER_SKU\", \"STANDARD_D2_V2\")\n",
    "\n",
    "if compute_name in ws.compute_targets:\n",
    "    compute_target = ws.compute_targets[compute_name]\n",
    "    if compute_target and type(compute_target) is AmlCompute:\n",
    "        print('found compute target. just use it. ' + compute_name)\n",
    "else:\n",
    "    print('creating a new compute target...')\n",
    "    provisioning_config = AmlCompute.provisioning_configuration(vm_size = vm_size,\n",
    "                                                                min_nodes = 0, \n",
    "                                                                max_nodes = 4,\n",
    "                                                                idle_seconds_before_scaledown=1800)\n",
    "\n",
    "    # create the cluster\n",
    "    compute_target = ComputeTarget.create(ws, compute_name, provisioning_config)\n",
    "    \n",
    "    # can poll for a minimum number of nodes and for a specific timeout. \n",
    "    # if no min node count is provided it will use the scale settings for the cluster\n",
    "    compute_target.wait_for_completion(show_output=True, min_node_count=None, timeout_in_minutes=20)\n",
    "    \n",
    "     # For a more detailed view of current AmlCompute status, use get_status()\n",
    "    print(compute_target.get_status().serialize())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create a datastore containing sample images\n",
    "The input dataset used for this notebook differs from a standard MNIST dataset in that it has been converted to PNG images to demonstrate use of files as inputs to Batch Inference. A sample of PNG-converted images of the MNIST dataset were take from [this repository](https://github.com/myleott/mnist_png).\n",
    "\n",
    "We have created a public blob container `sampledata` on an account named `pipelinedata`, containing these images from the MNIST dataset. In the next step, we create a datastore with the name `images_datastore`, which points to this blob container. In the call to `register_azure_blob_container` below, setting the `overwrite` flag to `True` overwrites any datastore that was created previously with that name. \n",
    "\n",
    "This step can be changed to point to your blob container by providing your own `datastore_name`, `container_name`, and `account_name`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.core.datastore import Datastore\n",
    "\n",
    "account_name = \"pipelinedata\"\n",
    "datastore_name = \"mnist_datastore\"\n",
    "container_name = \"sampledata\"\n",
    "\n",
    "mnist_data = Datastore.register_azure_blob_container(ws, \n",
    "                      datastore_name=datastore_name, \n",
    "                      container_name=container_name, \n",
    "                      account_name=account_name,\n",
    "                      overwrite=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, let's specify the default datastore for the outputs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def_data_store = ws.get_default_datastore()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create a FileDataset\n",
    "A [FileDataset](https://docs.microsoft.com/en-us/python/api/azureml-core/azureml.data.filedataset?view=azure-ml-py) references single or multiple files in your datastores or public urls. The files can be of any format. FileDataset provides you with the ability to download or mount the files to your compute. By creating a dataset, you create a reference to the data source location. If you applied any subsetting transformations to the dataset, they will be stored in the dataset as well. The data remains in its existing location, so no extra storage cost is incurred.\n",
    "You can use dataset objects as inputs. Register the datasets to the workspace if you want to reuse them later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.core.dataset import Dataset\n",
    "\n",
    "mnist_ds_name = 'mnist_sample_data'\n",
    "\n",
    "path_on_datastore = mnist_data.path('mnist')\n",
    "input_mnist_ds = Dataset.File.from_files(path=path_on_datastore, validate=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The input dataset can be specified as a pipeline parameter, so that you can pass in new data when rerun the PRS pipeline."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.data.dataset_consumption_config import DatasetConsumptionConfig\n",
    "from azureml.pipeline.core import PipelineParameter\n",
    "\n",
    "pipeline_param = PipelineParameter(name=\"mnist_param\", default_value=input_mnist_ds)\n",
    "input_mnist_ds_consumption = DatasetConsumptionConfig(\"minist_param_config\", pipeline_param).as_mount()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Intermediate/Output Data\n",
    "Intermediate data (or output of a Step) is represented by [PipelineData](https://docs.microsoft.com/en-us/python/api/azureml-pipeline-core/azureml.pipeline.core.pipelinedata?view=azure-ml-py) object. PipelineData can be produced by one step and consumed in another step by providing the PipelineData object as an output of one step and the input of one or more steps."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.pipeline.core import Pipeline, PipelineData\n",
    "\n",
    "output_dir = PipelineData(name=\"inferences\", datastore=def_data_store)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Download the Model\n",
    "\n",
    "Download and extract the model from https://pipelinedata.blob.core.windows.net/mnist-model/mnist-tf.tar.gz to \"models\" directory"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['mnist-tf.model.data-00000-of-00001',\n",
       " 'mnist-tf.model.index',\n",
       " 'mnist-tf.model.meta',\n",
       " 'saved_model.pb']"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import tarfile\n",
    "import urllib.request\n",
    "\n",
    "# create directory for model\n",
    "model_dir = 'models'\n",
    "if not os.path.isdir(model_dir):\n",
    "    os.mkdir(model_dir)\n",
    "\n",
    "url=\"https://pipelinedata.blob.core.windows.net/mnist-model/mnist-tf.tar.gz\"\n",
    "response = urllib.request.urlretrieve(url, \"model.tar.gz\")\n",
    "tar = tarfile.open(\"model.tar.gz\", \"r:gz\")\n",
    "tar.extractall(model_dir)\n",
    "\n",
    "os.listdir(model_dir)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Register the model with Workspace\n",
    "A registered model is a logical container for one or more files that make up your model. For example, if you have a model that's stored in multiple files, you can register them as a single model in the workspace. After you register the files, you can then download or deploy the registered model and receive all the files that you registered.\n",
    "\n",
    "Using tags, you can track useful information such as the name and version of the machine learning library used to train the model. Note that tags must be alphanumeric. Learn more about registering models [here](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-deploy-and-where#registermodel) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Registering model mnist-prs\n"
     ]
    }
   ],
   "source": [
    "from azureml.core.model import Model\n",
    "\n",
    "# register downloaded model \n",
    "model = Model.register(model_path=\"models/\",\n",
    "                       model_name=\"mnist-prs\", # this is the name the model is registered as\n",
    "                       tags={'pretrained': \"mnist\"},\n",
    "                       description=\"Mnist trained tensorflow model\",\n",
    "                       workspace=ws)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'mnist-prs:1'"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.id"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using your model to make batch predictions\n",
    "To use the model to make batch predictions, you need an **entry script** and a list of **dependencies**:\n",
    "\n",
    "#### An entry script\n",
    "This script accepts requests, scores the requests by using the model, and returns the results.\n",
    "- __init()__ - Typically this function loads the model into a global object. This function is run only once at the start of batch processing per worker node/process. Init method can make use of following environment variables (ParallelRunStep input):\n",
    "    1.\tAZUREML_BI_OUTPUT_PATH â€“ output folder path\n",
    "- __run(mini_batch)__ - The method to be parallelized. Each invocation will have one minibatch.<BR>\n",
    "__mini_batch__: Batch inference will invoke run method and pass either a list or Pandas DataFrame as an argument to the method. Each entry in min_batch will be - a filepath if input is a FileDataset, a Pandas DataFrame if input is a TabularDataset.<BR>\n",
    "__run__ method response: run() method should return a Pandas DataFrame or an array. For append_row output_action, these returned elements are appended into the common output file. For summary_only, the contents of the elements are ignored. For all output actions, each returned output element indicates one successful inference of input element in the input mini-batch.\n",
    "    User should make sure that enough data is included in inference result to map input to inference. Inference output will be written in output file and not guaranteed to be in order, user should use some key in the output to map it to input.\n",
    "    \n",
    "\n",
    "#### Dependencies\n",
    "Helper scripts or Python/Conda packages required to run the entry script."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "# Copyright (c) Microsoft. All rights reserved.\n",
      "# Licensed under the MIT license.\n",
      "\n",
      "import os\n",
      "import numpy as np\n",
      "import tensorflow as tf\n",
      "from PIL import Image\n",
      "from azureml.core import Model\n",
      "\n",
      "\n",
      "def init():\n",
      "    global g_tf_sess\n",
      "\n",
      "    # pull down model from workspace\n",
      "    model_path = Model.get_model_path(\"mnist-prs\")\n",
      "\n",
      "    # contruct graph to execute\n",
      "    tf.reset_default_graph()\n",
      "    saver = tf.train.import_meta_graph(os.path.join(model_path, 'mnist-tf.model.meta'))\n",
      "    g_tf_sess = tf.Session(config=tf.ConfigProto(device_count={'GPU': 0}))\n",
      "    saver.restore(g_tf_sess, os.path.join(model_path, 'mnist-tf.model'))\n",
      "\n",
      "\n",
      "def run(mini_batch):\n",
      "    print(f'run method start: {__file__}, run({mini_batch})')\n",
      "    resultList = []\n",
      "    in_tensor = g_tf_sess.graph.get_tensor_by_name(\"network/X:0\")\n",
      "    output = g_tf_sess.graph.get_tensor_by_name(\"network/output/MatMul:0\")\n",
      "\n",
      "    for image in mini_batch:\n",
      "        # prepare each image\n",
      "        data = Image.open(image)\n",
      "        np_im = np.array(data).reshape((1, 784))\n",
      "        # perform inference\n",
      "        inference_result = output.eval(feed_dict={in_tensor: np_im}, session=g_tf_sess)\n",
      "        # find best probability, and add to result list\n",
      "        best_result = np.argmax(inference_result)\n",
      "        resultList.append(\"{}: {}\".format(os.path.basename(image), best_result))\n",
      "\n",
      "    return resultList\n",
      "\n"
     ]
    }
   ],
   "source": [
    "scripts_folder = \"Code\"\n",
    "script_file = \"digit_identification.py\"\n",
    "\n",
    "# peek at contents\n",
    "with open(os.path.join(scripts_folder, script_file)) as inference_file:\n",
    "    print(inference_file.read())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Build and run the batch inference pipeline\n",
    "The data, models, and compute resource are now available. Let's put all these together in a pipeline."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  Specify the environment to run the script\n",
    "Specify the conda dependencies for your script. This will allow us to install pip packages as well as configure the inference environment.\n",
    "* Always include **azureml-core** and **azureml-dataset-runtime\\[fuse\\]** in the pip package list to make ParallelRunStep run properly.\n",
    "\n",
    "If you're using custom image (`batch_env.python.user_managed_dependencies = True`), you need to install the package to your image."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "'enabled' is deprecated. Please use the azureml.core.runconfig.DockerConfiguration object with the 'use_docker' param instead.\n"
     ]
    }
   ],
   "source": [
    "from azureml.core import Environment\n",
    "from azureml.core.runconfig import CondaDependencies, DEFAULT_CPU_IMAGE\n",
    "\n",
    "batch_conda_deps = CondaDependencies.create(pip_packages=[\"tensorflow==1.15.2\", \n",
    "                                                          \"pillow\", \n",
    "                                                          \"azureml-core\", \n",
    "                                                          \"azureml-dataset-runtime[fuse]\"])\n",
    "batch_env = Environment(name=\"batch_environment\")\n",
    "batch_env.python.conda_dependencies = batch_conda_deps\n",
    "batch_env.docker.enabled = True\n",
    "batch_env.docker.base_image = DEFAULT_CPU_IMAGE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  Create the configuration to wrap the inference script"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.pipeline.core import PipelineParameter\n",
    "from azureml.pipeline.steps import ParallelRunStep, ParallelRunConfig\n",
    "\n",
    "parallel_run_config = ParallelRunConfig(\n",
    "    source_directory=scripts_folder,\n",
    "    entry_script=script_file,\n",
    "    mini_batch_size=PipelineParameter(name=\"batch_size_param\", default_value=\"5\"),\n",
    "    error_threshold=10,\n",
    "    output_action=\"append_row\",\n",
    "    append_row_file_name=\"mnist_outputs.txt\",\n",
    "    environment=batch_env,\n",
    "    compute_target=compute_target,\n",
    "    process_count_per_node=PipelineParameter(name=\"process_count_param\", default_value=2),\n",
    "    node_count=2\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create the pipeline step\n",
    "Create the pipeline step using the script, environment configuration, and parameters. Specify the compute target you already attached to your workspace as the target of execution of the script. We will use ParallelRunStep to create the pipeline step."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "parallelrun_step = ParallelRunStep(\n",
    "    name=\"predict-digits-mnist\",\n",
    "    parallel_run_config=parallel_run_config,\n",
    "    inputs=[ input_mnist_ds_consumption ],\n",
    "    output=output_dir,\n",
    "    allow_reuse=False\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Run the pipeline\n",
    "At this point you can run the pipeline and examine the output it produced. The Experiment object is used to track the run of the pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Created step predict-digits-mnist [94471de5][df894262-e7d5-4e65-bfeb-e39dc6e45e0b], (This step will run and generate new outputs)\n",
      "Submitted PipelineRun c1a383cb-1311-4c4b-94bc-9848d18e5c96\n",
      "Link to Azure Machine Learning Portal: https://ml.azure.com/runs/c1a383cb-1311-4c4b-94bc-9848d18e5c96?wsid=/subscriptions/70b8f39e-8863-49f7-b6ba-34a80799550c/resourcegroups/azuremlworkshop-rg/workspaces/azuremlworkshop&tid=72f988bf-86f1-41af-91ab-2d7cd011db47\n"
     ]
    }
   ],
   "source": [
    "from azureml.core import Experiment\n",
    "\n",
    "pipeline = Pipeline(workspace=ws, steps=[parallelrun_step])\n",
    "experiment = Experiment(ws, 'Lab7-Pipelines')\n",
    "pipeline_run = experiment.submit(pipeline)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Monitor the run\n",
    "\n",
    "The pipeline run status could be checked in Azure Machine Learning portal (https://ml.azure.com). The link to the pipeline run could be retrieved by inspecting the `pipeline_run` object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table style=\"width:100%\"><tr><th>Experiment</th><th>Id</th><th>Type</th><th>Status</th><th>Details Page</th><th>Docs Page</th></tr><tr><td>Lab7-Pipelines</td><td>c1a383cb-1311-4c4b-94bc-9848d18e5c96</td><td>azureml.PipelineRun</td><td>Running</td><td><a href=\"https://ml.azure.com/runs/c1a383cb-1311-4c4b-94bc-9848d18e5c96?wsid=/subscriptions/70b8f39e-8863-49f7-b6ba-34a80799550c/resourcegroups/azuremlworkshop-rg/workspaces/azuremlworkshop&amp;tid=72f988bf-86f1-41af-91ab-2d7cd011db47\" target=\"_blank\" rel=\"noopener\">Link to Azure Machine Learning studio</a></td><td><a href=\"https://docs.microsoft.com/en-us/python/api/overview/azure/ml/intro?view=azure-ml-py\" target=\"_blank\" rel=\"noopener\">Link to Documentation</a></td></tr></table>"
      ],
      "text/plain": [
       "Run(Experiment: Lab7-Pipelines,\n",
       "Id: c1a383cb-1311-4c4b-94bc-9848d18e5c96,\n",
       "Type: azureml.PipelineRun,\n",
       "Status: Running)"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# This will output information of the pipeline run, including the link to the details page of portal.\n",
    "pipeline_run"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The pipeline is Finished at the moment ( 2021-08-27 08:43:22.487553 )\n"
     ]
    }
   ],
   "source": [
    "print(\"The pipeline is\", pipeline_run.get_status(), 'at the moment (', datetime.datetime.now(), ')')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Optional: View detailed logs (streaming) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PipelineRunId: c1a383cb-1311-4c4b-94bc-9848d18e5c96\n",
      "Link to Azure Machine Learning Portal: https://ml.azure.com/runs/c1a383cb-1311-4c4b-94bc-9848d18e5c96?wsid=/subscriptions/70b8f39e-8863-49f7-b6ba-34a80799550c/resourcegroups/azuremlworkshop-rg/workspaces/azuremlworkshop&tid=72f988bf-86f1-41af-91ab-2d7cd011db47\n",
      "\n",
      "PipelineRun Execution Summary\n",
      "==============================\n",
      "PipelineRun Status: Finished\n",
      "{'runId': 'c1a383cb-1311-4c4b-94bc-9848d18e5c96', 'status': 'Completed', 'startTimeUtc': '2021-08-27T08:04:25.650987Z', 'endTimeUtc': '2021-08-27T08:37:22.619677Z', 'properties': {'azureml.runsource': 'azureml.PipelineRun', 'runSource': 'SDK', 'runType': 'SDK', 'azureml.parameters': '{\"batch_size_param\":\"5\",\"process_count_param\":\"2\"}'}, 'inputDatasets': [], 'outputDatasets': [], 'logFiles': {'logs/azureml/executionlogs.txt': 'https://azuremlworksho7897742112.blob.core.windows.net/azureml/ExperimentRun/dcid.c1a383cb-1311-4c4b-94bc-9848d18e5c96/logs/azureml/executionlogs.txt?sv=2019-07-07&sr=b&sig=W%2BIJaZkCsPpb4ejSer%2Fw%2BndLEHsuJoDPaY03H%2BJBD9k%3D&st=2021-08-27T08%3A33%3A25Z&se=2021-08-27T16%3A43%3A25Z&sp=r', 'logs/azureml/stderrlogs.txt': 'https://azuremlworksho7897742112.blob.core.windows.net/azureml/ExperimentRun/dcid.c1a383cb-1311-4c4b-94bc-9848d18e5c96/logs/azureml/stderrlogs.txt?sv=2019-07-07&sr=b&sig=AezrX8o1Vwm6rpdsoXHFdgMCFax96oAn9p1ZPhhlcX8%3D&st=2021-08-27T08%3A33%3A25Z&se=2021-08-27T16%3A43%3A25Z&sp=r', 'logs/azureml/stdoutlogs.txt': 'https://azuremlworksho7897742112.blob.core.windows.net/azureml/ExperimentRun/dcid.c1a383cb-1311-4c4b-94bc-9848d18e5c96/logs/azureml/stdoutlogs.txt?sv=2019-07-07&sr=b&sig=fZzVU7HcDA9ZY%2BB6Z%2FBMLUHt7HEzIrQ08O1QRw0R%2FEU%3D&st=2021-08-27T08%3A33%3A25Z&se=2021-08-27T16%3A43%3A25Z&sp=r'}, 'submittedBy': 'Serge Retkowsky'}\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'Finished'"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Wait the run for completion and show output log to console\n",
    "pipeline_run.wait_for_completion(show_output=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### View the prediction results per input image\n",
    "In the digit_identification.py file above you can see that the ResultList with the filename and the prediction result gets returned. These are written to the DataStore specified in the PipelineData object as the output data, which in this case is called *inferences*. This containers the outputs from  all of the worker nodes used in the compute cluster. You can download this data to view the results ... below just filters to the first 10 rows"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Prediction has  1000  rows\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Filename</th>\n",
       "      <th>Prediction</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>102.png</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>103.png</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>104.png</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>105.png</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>106.png</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>143.png</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>144.png</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>145.png</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>146.png</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>147.png</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Filename  Prediction\n",
       "0  102.png           5\n",
       "1  103.png           4\n",
       "2  104.png           9\n",
       "3  105.png           9\n",
       "4  106.png           2\n",
       "5  143.png           1\n",
       "6  144.png           7\n",
       "7  145.png           1\n",
       "8  146.png           8\n",
       "9  147.png           2"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import tempfile\n",
    "\n",
    "batch_run = pipeline_run.find_step_run(parallelrun_step.name)[0]\n",
    "batch_output = batch_run.get_output_data(output_dir.name)\n",
    "\n",
    "target_dir = tempfile.mkdtemp()\n",
    "batch_output.download(local_path=target_dir)\n",
    "result_file = os.path.join(target_dir, batch_output.path_on_datastore, parallel_run_config.append_row_file_name)\n",
    "\n",
    "df = pd.read_csv(result_file, delimiter=\":\", header=None)\n",
    "df.columns = [\"Filename\", \"Prediction\"]\n",
    "print(\"Prediction has \", df.shape[0], \" rows\")\n",
    "df.head(10) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Resubmit a with different dataset\n",
    "Since we made the input a `PipelineParameter`, we can resubmit with a different dataset without having to create an entirely new experiment. We'll use the same datastore but use only a single image."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "path_on_datastore = mnist_data.path('mnist/0.png')\n",
    "single_image_ds = Dataset.File.from_files(path=path_on_datastore, validate=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Submitted PipelineRun ba67ad10-2e7b-451f-bf30-f957e21cf12b\n",
      "Link to Azure Machine Learning Portal: https://ml.azure.com/runs/ba67ad10-2e7b-451f-bf30-f957e21cf12b?wsid=/subscriptions/70b8f39e-8863-49f7-b6ba-34a80799550c/resourcegroups/azuremlworkshop-rg/workspaces/azuremlworkshop&tid=72f988bf-86f1-41af-91ab-2d7cd011db47\n"
     ]
    }
   ],
   "source": [
    "pipeline_run_2 = experiment.submit(pipeline, \n",
    "                                   pipeline_parameters={\"mnist_param\": single_image_ds, \n",
    "                                                        \"batch_size_param\": \"1\",\n",
    "                                                        \"process_count_param\": 1}\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table style=\"width:100%\"><tr><th>Experiment</th><th>Id</th><th>Type</th><th>Status</th><th>Details Page</th><th>Docs Page</th></tr><tr><td>Lab7-Pipelines</td><td>ba67ad10-2e7b-451f-bf30-f957e21cf12b</td><td>azureml.PipelineRun</td><td>Running</td><td><a href=\"https://ml.azure.com/runs/ba67ad10-2e7b-451f-bf30-f957e21cf12b?wsid=/subscriptions/70b8f39e-8863-49f7-b6ba-34a80799550c/resourcegroups/azuremlworkshop-rg/workspaces/azuremlworkshop&amp;tid=72f988bf-86f1-41af-91ab-2d7cd011db47\" target=\"_blank\" rel=\"noopener\">Link to Azure Machine Learning studio</a></td><td><a href=\"https://docs.microsoft.com/en-us/python/api/overview/azure/ml/intro?view=azure-ml-py\" target=\"_blank\" rel=\"noopener\">Link to Documentation</a></td></tr></table>"
      ],
      "text/plain": [
       "Run(Experiment: Lab7-Pipelines,\n",
       "Id: ba67ad10-2e7b-451f-bf30-f957e21cf12b,\n",
       "Type: azureml.PipelineRun,\n",
       "Status: Running)"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# This will output information of the pipeline run, including the link to the details page of portal.\n",
    "pipeline_run_2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PipelineRunId: ba67ad10-2e7b-451f-bf30-f957e21cf12b\n",
      "Link to Azure Machine Learning Portal: https://ml.azure.com/runs/ba67ad10-2e7b-451f-bf30-f957e21cf12b?wsid=/subscriptions/70b8f39e-8863-49f7-b6ba-34a80799550c/resourcegroups/azuremlworkshop-rg/workspaces/azuremlworkshop&tid=72f988bf-86f1-41af-91ab-2d7cd011db47\n",
      "PipelineRun Status: Running\n",
      "\n",
      "\n",
      "StepRunId: ba4d85cf-d82e-4c7c-bb78-325838b37271\n",
      "Link to Azure Machine Learning Portal: https://ml.azure.com/runs/ba4d85cf-d82e-4c7c-bb78-325838b37271?wsid=/subscriptions/70b8f39e-8863-49f7-b6ba-34a80799550c/resourcegroups/azuremlworkshop-rg/workspaces/azuremlworkshop&tid=72f988bf-86f1-41af-91ab-2d7cd011db47\n",
      "StepRun( predict-digits-mnist ) Status: NotStarted\n",
      "StepRun( predict-digits-mnist ) Status: Running\n",
      "\n",
      "Streaming azureml-logs/55_azureml-execution-tvmps_7f60c2ab84f5ed82a91597097db1e256bcdf0668aa1fbdf25cbbb7c0d9323244_d.txt\n",
      "========================================================================================================================\n",
      "2021-08-27T08:44:21Z Running following command: /bin/bash -c sudo blobfuse /mnt/batch/tasks/shared/LS_root/jobs/azuremlworkshop/azureml/ba4d85cf-d82e-4c7c-bb78-325838b37271/mounts/workspaceblobstore --tmp-path=/mnt/batch/tasks/shared/LS_root/jobs/azuremlworkshop/azureml/ba4d85cf-d82e-4c7c-bb78-325838b37271/caches/workspaceblobstore --file-cache-timeout-in-seconds=1000000 --cache-size-mb=88700 -o nonempty -o allow_other --config-file=/mnt/batch/tasks/shared/LS_root/jobs/azuremlworkshop/azureml/ba4d85cf-d82e-4c7c-bb78-325838b37271/configs/workspaceblobstore.cfg --log-level=LOG_WARNING\n",
      "2021-08-27T08:44:21Z Successfully mounted a/an Blobfuse File System at /mnt/batch/tasks/shared/LS_root/jobs/azuremlworkshop/azureml/ba4d85cf-d82e-4c7c-bb78-325838b37271/mounts/workspaceblobstore\n",
      "2021-08-27T08:44:22Z The vmsize standard_d2_v2 is not a GPU VM, skipping get GPU count by running nvidia-smi command.\n",
      "2021-08-27T08:44:22Z Starting output-watcher...\n",
      "2021-08-27T08:44:22Z IsDedicatedCompute == True, won't poll for Low Pri Preemption\n",
      "2021-08-27T08:44:22Z Executing 'Copy ACR Details file' on 10.0.0.6\n",
      "2021-08-27T08:44:22Z Executing 'Copy ACR Details file' on 10.0.0.5\n",
      "2021-08-27T08:44:22Z Copy ACR Details file succeeded on 10.0.0.5. Output: \n",
      ">>>   \n",
      ">>>   \n",
      "2021-08-27T08:44:23Z Copy ACR Details file succeeded on 10.0.0.6. Output: \n",
      ">>>   \n",
      "Login Succeeded\n",
      "Using default tag: latest\n",
      "latest: Pulling from azureml/azureml_1e5b59c0734bdc528077f509e1d397fe\n",
      "Digest: sha256:d0c5e40cf440c619e721faa614dc94fe56cb6c41768532d169421cbe2288edc7\n",
      "Status: Image is up to date for viennaglobal.azurecr.io/azureml/azureml_1e5b59c0734bdc528077f509e1d397fe:latest\n",
      "viennaglobal.azurecr.io/azureml/azureml_1e5b59c0734bdc528077f509e1d397fe:latest\n",
      "2021-08-27T08:44:24Z The vmsize standard_d2_v2 is not a GPU VM, skipping get GPU count by running nvidia-smi command.\n",
      "2021-08-27T08:44:24Z Check if container ba4d85cf-d82e-4c7c-bb78-325838b37271_DataSidecar already exist exited with 0, \n",
      "\n",
      "ff849ba895a37eddb6dde8feadc33db3d0e196c3d87c737e98d05952eaec1a41\n",
      "2021-08-27T08:44:24Z Parameters for containerSetup task: useDetonationChamer set to false and sshRequired set to false \n",
      "2021-08-27T08:44:24Z containerSetup task cmd: [/mnt/batch/tasks/startup/wd/hosttools -task=containerSetup -traceContext=00-5e0eb1f38c2f5f2605f68e5788656cc1-3eafcc92cb20f0f6-01 -sshRequired=false] \n",
      "2021/08/27 08:44:24 Starting App Insight Logger for task:  containerSetup\n",
      "2021/08/27 08:44:24 Version: 3.0.01695.0004 Branch: .SourceBranch Commit: d8186cb\n",
      "2021/08/27 08:44:24 Entered ContainerSetupTask - Preparing infiniband\n",
      "2021/08/27 08:44:24 Starting infiniband setup\n",
      "2021/08/27 08:44:24 Python Version found is Python 3.7.9\n",
      "\n",
      "2021/08/27 08:44:24 Returning Python Version as 3.7\n",
      "2021-08-27T08:44:24Z VMSize: standard_d2_v2, Host: runtime-gen1-ubuntu18, Container: ubuntu-16.04\n",
      "2021-08-27T08:44:24Z Not setting up Infiniband in Container\n",
      "2021/08/27 08:44:24 VMSize: standard_d2_v2, Host: runtime-gen1-ubuntu18, Container: ubuntu-16.04\n",
      "2021/08/27 08:44:24 VMSize: standard_d2_v2, Host: runtime-gen1-ubuntu18, Container: ubuntu-16.04\n",
      "2021/08/27 08:44:24 /dev/infiniband/uverbs0 found (implying presence of InfiniBand)?: false\n",
      "2021/08/27 08:44:24 Not setting up Infiniband in Container\n",
      "2021/08/27 08:44:24 Not setting up Infiniband in Container\n",
      "2021/08/27 08:44:24 Python Version found is Python 3.7.9\n",
      "\n",
      "2021/08/27 08:44:24 Returning Python Version as 3.7\n",
      "2021/08/27 08:44:24 sshd inside container not required for job, skipping setup.\n",
      "2021/08/27 08:44:24 All App Insights Logs was sent successfully or the close timeout of 10 was reached\n",
      "2021/08/27 08:44:24 App Insight Client has already been closed\n",
      "2021/08/27 08:44:24 Not exporting to RunHistory as the exporter is either stopped or there is no data.\n",
      "Stopped: false\n",
      "OriginalData: 1\n",
      "FilteredData: 0.\n",
      "2021-08-27T08:44:24Z Starting docker container succeeded.\n",
      "2021-08-27T08:44:25Z The vmsize standard_d2_v2 is not a GPU VM, skipping get GPU count by running nvidia-smi command.\n",
      "\n",
      "Streaming azureml-logs/65_job_prep-tvmps_7f60c2ab84f5ed82a91597097db1e256bcdf0668aa1fbdf25cbbb7c0d9323244_d.txt\n",
      "===============================================================================================================\n",
      "[2021-08-27T08:44:27.685188] Entering job preparation.\n",
      "[2021-08-27T08:44:29.138933] Starting job preparation.\n",
      "[2021-08-27T08:44:29.138983] Extracting the control code.\n",
      "[2021-08-27T08:44:29.139479] Starting extract_project.\n",
      "[2021-08-27T08:44:29.139531] Starting to extract zip file.\n",
      "[2021-08-27T08:44:29.159578] Finished extracting zip file.\n",
      "[2021-08-27T08:44:29.163124] Using urllib.request Python 3.0 or later\n",
      "[2021-08-27T08:44:29.163209] Start fetching snapshots.\n",
      "[2021-08-27T08:44:29.163284] Start fetching snapshot.\n",
      "[2021-08-27T08:44:29.163310] Retrieving project from snapshot: 184f96ee-f9c4-4305-aef5-f34b2630313c\n",
      "Starting the daemon thread to refresh tokens in background for process with pid = 51\n",
      "[2021-08-27T08:44:29.436942] Finished fetching snapshot.\n",
      "[2021-08-27T08:44:29.436982] Start fetching snapshot.\n",
      "[2021-08-27T08:44:29.437004] Retrieving project from snapshot: 00b12447-c006-4c00-a382-d5db29734052\n",
      "[2021-08-27T08:44:37.397150] Finished fetching snapshot.\n",
      "[2021-08-27T08:44:37.397203] Finished fetching snapshots.\n",
      "[2021-08-27T08:44:37.397213] Finished extract_project.\n",
      "[2021-08-27T08:44:37.397621] Finished fetching and extracting the control code.\n",
      "[2021-08-27T08:44:37.408114] Start run_history_prep.\n",
      "[2021-08-27T08:44:37.416130] Job preparation is complete.\n",
      "[2021-08-27T08:44:37.416273] Entering Data Context Managers in Sidecar\n",
      "[2021-08-27T08:44:37.417236] Running Sidecar prep cmd...\n",
      "[2021-08-27T08:44:37.867711] INFO azureml.sidecar.sidecar: Received task: enter_contexts. Running on Linux at /mnt/hostfs/mnt/batch/tasks/shared/LS_root/jobs/azuremlworkshop/azureml/ba4d85cf-d82e-4c7c-bb78-325838b37271/wd/azureml/ba4d85cf-d82e-4c7c-bb78-325838b37271\n",
      "[2021-08-27T08:44:37.869302] INFO azureml.sidecar.sidecar: Invoking \"enter_contexts\" task with Context Managers: {\"context_managers\": [\"Dataset:context_managers.Datasets\", \"DataStoreCopy:context_managers.DataStores\"]}\n",
      "Enter __enter__ of DatasetContextManager\n",
      "SDK version: azureml-core==1.32.0 azureml-dataprep==2.20.1. Session id: 65e117e1-8493-4c2f-a983-566a485902f5. Run id: ba4d85cf-d82e-4c7c-bb78-325838b37271.\n",
      "Processing 'minist_param_config'.\n",
      "Mode: 'mount'.\n",
      "Path on compute is specified: 'False'.\n",
      "Processing dataset FileDataset\n",
      "{\n",
      "  \"source\": [\n",
      "    \"('mnist_datastore', 'mnist/0.png')\"\n",
      "  ],\n",
      "  \"definition\": [\n",
      "    \"GetDatastoreFiles\"\n",
      "  ],\n",
      "  \"registration\": {\n",
      "    \"id\": \"936dd865-fa43-4c2a-85b1-0cf530f6fa0e\",\n",
      "    \"name\": null,\n",
      "    \"version\": null,\n",
      "    \"workspace\": \"Workspace.create(name='azuremlworkshop', subscription_id='70b8f39e-8863-49f7-b6ba-34a80799550c', resource_group='azuremlworkshop-rg')\"\n",
      "  }\n",
      "}\n",
      "Mounting minist_param_config to /mnt/hostfs/mnt/batch/tasks/shared/LS_root/jobs/azuremlworkshop/azureml/ba4d85cf-d82e-4c7c-bb78-325838b37271/wd/minist_param_config_936dd865-fa43-4c2a-85b1-0cf530f6fa0e.\n",
      "Mounted minist_param_config to /mnt/hostfs/mnt/batch/tasks/shared/LS_root/jobs/azuremlworkshop/azureml/ba4d85cf-d82e-4c7c-bb78-325838b37271/wd/minist_param_config_936dd865-fa43-4c2a-85b1-0cf530f6fa0e as single file.\n",
      "Exit __enter__ of DatasetContextManager\n",
      "Set Dataset minist_param_config's target path to /mnt/batch/tasks/shared/LS_root/jobs/azuremlworkshop/azureml/ba4d85cf-d82e-4c7c-bb78-325838b37271/wd/minist_param_config_936dd865-fa43-4c2a-85b1-0cf530f6fa0e/0.png\n",
      "Sidecar adding paths_to_bind: ['/mnt/batch/tasks/shared/LS_root/jobs/azuremlworkshop/azureml/ba4d85cf-d82e-4c7c-bb78-325838b37271/wd/tmp_5zqn_91:/tmp/5d09dcc1-e7fe-467b-94b8-6f6cafcdb1a7/8961e04a-c8ed-4cbc-8ced-f40e9a79afee']\n",
      "Acquired lockfile /tmp/ba4d85cf-d82e-4c7c-bb78-325838b37271-datastore.lock to downloading input data references\n",
      "[2021-08-27T08:44:47.602333] INFO azureml.sidecar.task.enter_contexts: Entered Context Managers\n",
      "[2021-08-27T08:44:48.055798] Ran Sidecar prep cmd.\n",
      "[2021-08-27T08:44:48.055908] Running Context Managers in Sidecar complete.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Streaming azureml-logs/70_driver_log.txt\n",
      "========================================\n",
      "2021/08/27 08:45:00 Starting App Insight Logger for task:  runTaskLet\n",
      "2021/08/27 08:45:00 Version: 3.0.01695.0004 Branch: .SourceBranch Commit: d8186cb\n",
      "2021/08/27 08:45:00 Attempt 1 of http call to http://10.0.0.5:16384/sendlogstoartifacts/info\n",
      "2021/08/27 08:45:00 Send process info logs to master server succeeded\n",
      "2021/08/27 08:45:00 Attempt 1 of http call to http://10.0.0.5:16384/sendlogstoartifacts/status\n",
      "2021/08/27 08:45:00 Send process info logs to master server succeeded\n",
      "[2021-08-27T08:45:00.464406] Entering context manager injector.\n",
      "[2021-08-27T08:45:01.094694] context_manager_injector.py Command line Options: Namespace(inject=['ProjectPythonPath:context_managers.ProjectPythonPath', 'Dataset:context_managers.Datasets', 'RunHistory:context_managers.RunHistory', 'TrackUserError:context_managers.TrackUserError', 'UserExceptions:context_managers.UserExceptions'], invocation=['driver/amlbi_main.py', '--client_sdk_version', '1.33.0', '--scoring_module_name', 'digit_identification.py', '--mini_batch_size', '1', '--error_threshold', '10', '--output_action', 'append_row', '--logging_level', 'INFO', '--run_invocation_timeout', '60', '--run_max_try', '3', '--create_snapshot_at_runtime', 'True', '--append_row_file_name', 'mnist_outputs.txt', '--output', '/mnt/batch/tasks/shared/LS_root/jobs/azuremlworkshop/azureml/ba4d85cf-d82e-4c7c-bb78-325838b37271/mounts/workspaceblobstore/azureml/ba4d85cf-d82e-4c7c-bb78-325838b37271/inferences', '--process_count_per_node', '1', '--input_fds_0', 'minist_param_config', '--input_pipeline_param_0', 'DatasetConsumptionConfig:minist_param_config'])\n",
      "Script type = None\n",
      "[2021-08-27T08:45:01.100148] Entering Run History Context Manager.\n",
      "[2021-08-27T08:45:01.951080] Current directory: /mnt/batch/tasks/shared/LS_root/jobs/azuremlworkshop/azureml/ba4d85cf-d82e-4c7c-bb78-325838b37271/wd/azureml/ba4d85cf-d82e-4c7c-bb78-325838b37271\n",
      "[2021-08-27T08:45:01.951892] Preparing to call script [driver/amlbi_main.py] with arguments:['--client_sdk_version', '1.33.0', '--scoring_module_name', 'digit_identification.py', '--mini_batch_size', '1', '--error_threshold', '10', '--output_action', 'append_row', '--logging_level', 'INFO', '--run_invocation_timeout', '60', '--run_max_try', '3', '--create_snapshot_at_runtime', 'True', '--append_row_file_name', 'mnist_outputs.txt', '--output', '/mnt/batch/tasks/shared/LS_root/jobs/azuremlworkshop/azureml/ba4d85cf-d82e-4c7c-bb78-325838b37271/mounts/workspaceblobstore/azureml/ba4d85cf-d82e-4c7c-bb78-325838b37271/inferences', '--process_count_per_node', '1', '--input_fds_0', 'minist_param_config', '--input_pipeline_param_0', '$minist_param_config']\n",
      "[2021-08-27T08:45:01.951929] After variable expansion, calling script [driver/amlbi_main.py] with arguments:['--client_sdk_version', '1.33.0', '--scoring_module_name', 'digit_identification.py', '--mini_batch_size', '1', '--error_threshold', '10', '--output_action', 'append_row', '--logging_level', 'INFO', '--run_invocation_timeout', '60', '--run_max_try', '3', '--create_snapshot_at_runtime', 'True', '--append_row_file_name', 'mnist_outputs.txt', '--output', '/mnt/batch/tasks/shared/LS_root/jobs/azuremlworkshop/azureml/ba4d85cf-d82e-4c7c-bb78-325838b37271/mounts/workspaceblobstore/azureml/ba4d85cf-d82e-4c7c-bb78-325838b37271/inferences', '--process_count_per_node', '1', '--input_fds_0', 'minist_param_config', '--input_pipeline_param_0', '/mnt/batch/tasks/shared/LS_root/jobs/azuremlworkshop/azureml/ba4d85cf-d82e-4c7c-bb78-325838b37271/wd/minist_param_config_936dd865-fa43-4c2a-85b1-0cf530f6fa0e/0.png']\n",
      "\n",
      "2021/08/27 08:45:05 Not exporting to RunHistory as the exporter is either stopped or there is no data.\n",
      "Stopped: false\n",
      "OriginalData: 1\n",
      "FilteredData: 0.\n",
      "\n",
      "Streaming azureml-logs/75_job_post-tvmps_7f60c2ab84f5ed82a91597097db1e256bcdf0668aa1fbdf25cbbb7c0d9323244_d.txt\n",
      "===============================================================================================================\n",
      "[2021-08-27T08:45:49.758901] Entering job release\n",
      "[2021-08-27T08:45:50.748703] Starting job release\n",
      "[2021-08-27T08:45:50.749477] Logging experiment finalizing status in history service.\n",
      "Starting the daemon thread to refresh tokens in background for process with pid = 263\n",
      "[2021-08-27T08:45:50.750165] job release stage : upload_datastore starting...\n",
      "[2021-08-27T08:45:50.753489] job release stage : start importing azureml.history._tracking in run_history_release.[2021-08-27T08:45:50.764152] job release stage : copy_batchai_cached_logs starting...\n",
      "\n",
      "[2021-08-27T08:45:50.764867] Entering context manager injector.\n",
      "[2021-08-27T08:45:50.764752] job release stage : copy_batchai_cached_logs completed...[2021-08-27T08:45:50.765818] job release stage : execute_job_release starting...\n",
      "\n",
      "[2021-08-27T08:45:50.779674] job release stage : upload_datastore completed...\n",
      "[2021-08-27T08:45:50.839876] job release stage : send_run_telemetry starting...\n",
      "[2021-08-27T08:45:50.864496] get vm size and vm region successfully.\n",
      "[2021-08-27T08:45:50.878079] get compute meta data successfully.\n",
      "[2021-08-27T08:45:51.023406] job release stage : execute_job_release completed...\n",
      "[2021-08-27T08:45:51.073812] post artifact meta request successfully.\n",
      "[2021-08-27T08:45:51.103896] upload compute record artifact successfully.\n",
      "[2021-08-27T08:45:51.103995] job release stage : send_run_telemetry completed...\n",
      "[2021-08-27T08:45:51.104322] Running in AzureML-Sidecar, starting to exit user context managers...\n",
      "[2021-08-27T08:45:51.104439] Running Sidecar release cmd...\n",
      "[2021-08-27T08:45:51.127321] INFO azureml.sidecar.sidecar: Received task: exit_contexts. Running on Linux at /mnt/hostfs/mnt/batch/tasks/shared/LS_root/jobs/azuremlworkshop/azureml/ba4d85cf-d82e-4c7c-bb78-325838b37271/wd/azureml/ba4d85cf-d82e-4c7c-bb78-325838b37271\n",
      "Enter __exit__ of DatasetContextManager\n",
      "Unmounting /mnt/hostfs/mnt/batch/tasks/shared/LS_root/jobs/azuremlworkshop/azureml/ba4d85cf-d82e-4c7c-bb78-325838b37271/wd/minist_param_config_936dd865-fa43-4c2a-85b1-0cf530f6fa0e.\n",
      "fuse: failed to unmount /mnt/hostfs/mnt/batch/tasks/shared/LS_root/jobs/azuremlworkshop/azureml/ba4d85cf-d82e-4c7c-bb78-325838b37271/wd/minist_param_config_936dd865-fa43-4c2a-85b1-0cf530f6fa0e: Invalid argument\n",
      "Finishing unmounting /mnt/hostfs/mnt/batch/tasks/shared/LS_root/jobs/azuremlworkshop/azureml/ba4d85cf-d82e-4c7c-bb78-325838b37271/wd/minist_param_config_936dd865-fa43-4c2a-85b1-0cf530f6fa0e.\n",
      "Exit __exit__ of DatasetContextManager\n",
      "[2021-08-27T08:45:51.168712] Removing absolute paths from host...\n",
      "[2021-08-27T08:45:51.169006] INFO azureml.sidecar.task.exit_contexts: Exited Context Managers\n",
      "[2021-08-27T08:45:51.385241] Ran Sidecar release cmd.\n",
      "[2021-08-27T08:45:51.385344] Job release is complete\n",
      "\n",
      "StepRun(predict-digits-mnist) Execution Summary\n",
      "================================================\n",
      "StepRun( predict-digits-mnist ) Status: Finished\n",
      "{'runId': 'ba4d85cf-d82e-4c7c-bb78-325838b37271', 'target': 'pipelines342120', 'status': 'Completed', 'startTimeUtc': '2021-08-27T08:44:17.581958Z', 'endTimeUtc': '2021-08-27T08:46:11.88447Z', 'properties': {'ContentSnapshotId': '184f96ee-f9c4-4305-aef5-f34b2630313c', 'StepType': 'PythonScriptStep', 'ComputeTargetType': 'AmlCompute', 'azureml.moduleid': 'df894262-e7d5-4e65-bfeb-e39dc6e45e0b', 'azureml.runsource': 'azureml.StepRun', 'azureml.nodeid': '94471de5', 'azureml.pipelinerunid': 'ba67ad10-2e7b-451f-bf30-f957e21cf12b', '_azureml.ComputeTargetType': 'amlcompute', 'ProcessInfoFile': 'azureml-logs/process_info.json', 'ProcessStatusFile': 'azureml-logs/process_status.json', 'azureml.parallelrunstep': 'true'}, 'inputDatasets': [{'dataset': {'id': '936dd865-fa43-4c2a-85b1-0cf530f6fa0e'}, 'consumptionDetails': {'type': 'RunInput', 'inputName': 'minist_param_config', 'mechanism': 'Mount'}}], 'outputDatasets': [], 'runDefinition': {'script': 'driver/amlbi_main.py', 'command': '', 'useAbsolutePath': False, 'arguments': ['--client_sdk_version', '1.33.0', '--scoring_module_name', 'digit_identification.py', '--mini_batch_size', '$AML_PARAMETER_batch_size_param', '--error_threshold', '10', '--output_action', 'append_row', '--logging_level', 'INFO', '--run_invocation_timeout', '60', '--run_max_try', '3', '--create_snapshot_at_runtime', 'True', '--append_row_file_name', 'mnist_outputs.txt', '--output', '$AZUREML_DATAREFERENCE_inferences', '--process_count_per_node', '$AML_PARAMETER_process_count_param', '--input_fds_0', 'minist_param_config', '--input_pipeline_param_0', 'DatasetConsumptionConfig:minist_param_config'], 'sourceDirectoryDataStore': None, 'framework': 'Python', 'communicator': 'None', 'target': 'pipelines342120', 'dataReferences': {'inferences': {'dataStoreName': 'workspaceblobstore', 'mode': 'Mount', 'pathOnDataStore': 'azureml/ba4d85cf-d82e-4c7c-bb78-325838b37271/inferences', 'pathOnCompute': None, 'overwrite': False}}, 'data': {'minist_param_config': {'dataLocation': {'dataset': {'id': '936dd865-fa43-4c2a-85b1-0cf530f6fa0e', 'name': None, 'version': None}, 'dataPath': None, 'uri': None}, 'mechanism': 'Mount', 'environmentVariableName': 'minist_param_config', 'pathOnCompute': None, 'overwrite': False, 'options': None}}, 'outputData': {}, 'datacaches': [], 'jobName': None, 'maxRunDurationSeconds': None, 'nodeCount': 2, 'priority': None, 'credentialPassthrough': False, 'identity': None, 'environment': {'name': 'batch_environment', 'version': 'Autosave_2021-08-27T08:05:43Z_02d3de0f', 'python': {'interpreterPath': 'python', 'userManagedDependencies': False, 'condaDependencies': {'channels': ['anaconda', 'conda-forge'], 'dependencies': ['python=3.6.2', {'pip': ['tensorflow==1.15.2', 'pillow', 'azureml-core~=1.33.0', 'azureml-dataset-runtime[fuse]~=1.33.0']}], 'name': 'azureml_64e4ccbcbe9f8c4ad4f0736836a8d183'}, 'baseCondaEnvironment': None}, 'environmentVariables': {'EXAMPLE_ENV_VAR': 'EXAMPLE_VALUE'}, 'docker': {'baseImage': 'mcr.microsoft.com/azureml/openmpi3.1.2-ubuntu18.04:20210714.v1', 'platform': {'os': 'Linux', 'architecture': 'amd64'}, 'baseDockerfile': None, 'baseImageRegistry': {'address': None, 'username': None, 'password': None}, 'enabled': True, 'arguments': []}, 'spark': {'repositories': [], 'packages': [], 'precachePackages': True}, 'inferencingStackVersion': None}, 'history': {'outputCollection': True, 'directoriesToWatch': ['logs'], 'enableMLflowTracking': True, 'snapshotProject': True}, 'spark': {'configuration': {'spark.app.name': 'Azure ML Experiment', 'spark.yarn.maxAppAttempts': '1'}}, 'parallelTask': {'maxRetriesPerWorker': 0, 'workerCountPerNode': 1, 'terminalExitCodes': None, 'configuration': {}}, 'amlCompute': {'name': None, 'vmSize': None, 'retainCluster': False, 'clusterMaxNodeCount': 1}, 'aiSuperComputer': {'instanceType': 'D2', 'imageVersion': 'pytorch-1.7.0', 'location': None, 'aiSuperComputerStorageData': None, 'interactive': False, 'scalePolicy': None, 'virtualClusterArmId': None, 'tensorboardLogDirectory': None, 'sshPublicKey': None, 'enableAzmlInt': True, 'priority': 'Medium', 'slaTier': 'Standard', 'userAlias': None}, 'kubernetesCompute': {'instanceType': None}, 'tensorflow': {'workerCount': 1, 'parameterServerCount': 1}, 'mpi': {'processCountPerNode': 1}, 'pyTorch': {'communicationBackend': 'nccl', 'processCount': None}, 'hdi': {'yarnDeployMode': 'Cluster'}, 'containerInstance': {'region': None, 'cpuCores': 2.0, 'memoryGb': 3.5}, 'exposedPorts': None, 'docker': {'useDocker': True, 'sharedVolumes': True, 'shmSize': '2g', 'arguments': []}, 'cmk8sCompute': {'configuration': {}}, 'commandReturnCodeConfig': {'returnCode': 'Zero', 'successfulReturnCodes': []}, 'environmentVariables': {'AML_PARAMETER_batch_size_param': '1', 'AML_PARAMETER_process_count_param': '1'}, 'applicationEndpoints': {}, 'parameters': [], 'dataBricks': {'workers': 0, 'minimumWorkerCount': 0, 'maxMumWorkerCount': 0, 'sparkVersion': '4.0.x-scala2.11', 'nodeTypeId': 'Standard_D3_v2', 'sparkConf': {}, 'sparkEnvVars': {}, 'instancePoolId': None, 'timeoutSeconds': 0, 'jarLibraries': [], 'eggLibraries': [], 'whlLibraries': [], 'pypiLibraries': [], 'rCranLibraries': [], 'mavenLibraries': []}}, 'logFiles': {'azureml-logs/55_azureml-execution-tvmps_7f60c2ab84f5ed82a91597097db1e256bcdf0668aa1fbdf25cbbb7c0d9323244_d.txt': 'https://azuremlworksho7897742112.blob.core.windows.net/azureml/ExperimentRun/dcid.ba4d85cf-d82e-4c7c-bb78-325838b37271/azureml-logs/55_azureml-execution-tvmps_7f60c2ab84f5ed82a91597097db1e256bcdf0668aa1fbdf25cbbb7c0d9323244_d.txt?sv=2019-07-07&sr=b&sig=jC2lEwYp2MoGHdQ3DUnIHmS9mMFc%2BhNqAp04ERnUZRw%3D&st=2021-08-27T08%3A35%3A58Z&se=2021-08-27T16%3A45%3A58Z&sp=r', 'azureml-logs/55_azureml-execution-tvmps_f7393ee5d1f97ad75e43509d348b2677c577350c27484eaa7b0d5ef01cbfa2e6_d.txt': 'https://azuremlworksho7897742112.blob.core.windows.net/azureml/ExperimentRun/dcid.ba4d85cf-d82e-4c7c-bb78-325838b37271/azureml-logs/55_azureml-execution-tvmps_f7393ee5d1f97ad75e43509d348b2677c577350c27484eaa7b0d5ef01cbfa2e6_d.txt?sv=2019-07-07&sr=b&sig=k5vc1jv17qt6LPU4vubevC17lgkvY57zT%2BnAXvdSM9I%3D&st=2021-08-27T08%3A35%3A58Z&se=2021-08-27T16%3A45%3A58Z&sp=r', 'azureml-logs/65_job_prep-tvmps_7f60c2ab84f5ed82a91597097db1e256bcdf0668aa1fbdf25cbbb7c0d9323244_d.txt': 'https://azuremlworksho7897742112.blob.core.windows.net/azureml/ExperimentRun/dcid.ba4d85cf-d82e-4c7c-bb78-325838b37271/azureml-logs/65_job_prep-tvmps_7f60c2ab84f5ed82a91597097db1e256bcdf0668aa1fbdf25cbbb7c0d9323244_d.txt?sv=2019-07-07&sr=b&sig=%2F9CSLTgdKaLescpP13%2FCeMGBBKiCGb9Ly05vEO%2BmdcA%3D&st=2021-08-27T08%3A35%3A58Z&se=2021-08-27T16%3A45%3A58Z&sp=r', 'azureml-logs/65_job_prep-tvmps_f7393ee5d1f97ad75e43509d348b2677c577350c27484eaa7b0d5ef01cbfa2e6_d.txt': 'https://azuremlworksho7897742112.blob.core.windows.net/azureml/ExperimentRun/dcid.ba4d85cf-d82e-4c7c-bb78-325838b37271/azureml-logs/65_job_prep-tvmps_f7393ee5d1f97ad75e43509d348b2677c577350c27484eaa7b0d5ef01cbfa2e6_d.txt?sv=2019-07-07&sr=b&sig=p%2B8Mk4Sv6gRgU7vMd8skF6f9WYXpqe83zdPGI0cHgKc%3D&st=2021-08-27T08%3A35%3A58Z&se=2021-08-27T16%3A45%3A58Z&sp=r', 'azureml-logs/70_driver_log.txt': 'https://azuremlworksho7897742112.blob.core.windows.net/azureml/ExperimentRun/dcid.ba4d85cf-d82e-4c7c-bb78-325838b37271/azureml-logs/70_driver_log.txt?sv=2019-07-07&sr=b&sig=kgS4NW%2B%2FWKzu7nhMypu%2FhSuHnq80YfQu3eCIAjhgDsM%3D&st=2021-08-27T08%3A35%3A58Z&se=2021-08-27T16%3A45%3A58Z&sp=r', 'azureml-logs/75_job_post-tvmps_7f60c2ab84f5ed82a91597097db1e256bcdf0668aa1fbdf25cbbb7c0d9323244_d.txt': 'https://azuremlworksho7897742112.blob.core.windows.net/azureml/ExperimentRun/dcid.ba4d85cf-d82e-4c7c-bb78-325838b37271/azureml-logs/75_job_post-tvmps_7f60c2ab84f5ed82a91597097db1e256bcdf0668aa1fbdf25cbbb7c0d9323244_d.txt?sv=2019-07-07&sr=b&sig=X7ehMDNth6eLrld4ylOlZUsrsNz5grNk1LjSVTfFYKg%3D&st=2021-08-27T08%3A35%3A58Z&se=2021-08-27T16%3A45%3A58Z&sp=r', 'azureml-logs/75_job_post-tvmps_f7393ee5d1f97ad75e43509d348b2677c577350c27484eaa7b0d5ef01cbfa2e6_d.txt': 'https://azuremlworksho7897742112.blob.core.windows.net/azureml/ExperimentRun/dcid.ba4d85cf-d82e-4c7c-bb78-325838b37271/azureml-logs/75_job_post-tvmps_f7393ee5d1f97ad75e43509d348b2677c577350c27484eaa7b0d5ef01cbfa2e6_d.txt?sv=2019-07-07&sr=b&sig=e%2F9PP8FMKRV74ARhgdx%2Fx%2F%2F%2FNHRDr2c%2FZ0NHmPQjpn8%3D&st=2021-08-27T08%3A35%3A58Z&se=2021-08-27T16%3A45%3A58Z&sp=r', 'azureml-logs/process_info.json': 'https://azuremlworksho7897742112.blob.core.windows.net/azureml/ExperimentRun/dcid.ba4d85cf-d82e-4c7c-bb78-325838b37271/azureml-logs/process_info.json?sv=2019-07-07&sr=b&sig=yWh2MRbULgP1bHyteNDXqn6%2Fx9MTgDaMAppQtYHUr7o%3D&st=2021-08-27T08%3A35%3A58Z&se=2021-08-27T16%3A45%3A58Z&sp=r', 'azureml-logs/process_status.json': 'https://azuremlworksho7897742112.blob.core.windows.net/azureml/ExperimentRun/dcid.ba4d85cf-d82e-4c7c-bb78-325838b37271/azureml-logs/process_status.json?sv=2019-07-07&sr=b&sig=fs4%2Bs%2FHUxUrfBUOzYBfiYRccfISKTNjLMrHNf%2Fq6dFA%3D&st=2021-08-27T08%3A35%3A58Z&se=2021-08-27T16%3A45%3A58Z&sp=r', 'logs/azureml/104_azureml.log': 'https://azuremlworksho7897742112.blob.core.windows.net/azureml/ExperimentRun/dcid.ba4d85cf-d82e-4c7c-bb78-325838b37271/logs/azureml/104_azureml.log?sv=2019-07-07&sr=b&sig=cwCznNNps9BiFsSEeOYfgBvUBEaoCuD6VwFfyh2mDxk%3D&st=2021-08-27T08%3A35%3A58Z&se=2021-08-27T16%3A45%3A58Z&sp=r', 'logs/azureml/89_azureml.log': 'https://azuremlworksho7897742112.blob.core.windows.net/azureml/ExperimentRun/dcid.ba4d85cf-d82e-4c7c-bb78-325838b37271/logs/azureml/89_azureml.log?sv=2019-07-07&sr=b&sig=OD2jAnCHVW%2BRQEjXu1oOEMXpvh626GfU6LR1TumPtVA%3D&st=2021-08-27T08%3A35%3A58Z&se=2021-08-27T16%3A45%3A58Z&sp=r', 'logs/azureml/dataprep/backgroundProcess.log': 'https://azuremlworksho7897742112.blob.core.windows.net/azureml/ExperimentRun/dcid.ba4d85cf-d82e-4c7c-bb78-325838b37271/logs/azureml/dataprep/backgroundProcess.log?sv=2019-07-07&sr=b&sig=LrcN9Jx%2FJo3T4ZptKMEedJxbFV1r0OxJZwdEtAq6SyQ%3D&st=2021-08-27T08%3A35%3A58Z&se=2021-08-27T16%3A45%3A58Z&sp=r', 'logs/azureml/dataprep/backgroundProcess_Telemetry.log': 'https://azuremlworksho7897742112.blob.core.windows.net/azureml/ExperimentRun/dcid.ba4d85cf-d82e-4c7c-bb78-325838b37271/logs/azureml/dataprep/backgroundProcess_Telemetry.log?sv=2019-07-07&sr=b&sig=VloSLs7GKHPkX9SSA%2FxDytP93aU%2Bk4wBWvSIlcInJt0%3D&st=2021-08-27T08%3A35%3A58Z&se=2021-08-27T16%3A45%3A58Z&sp=r', 'logs/azureml/executionlogs.txt': 'https://azuremlworksho7897742112.blob.core.windows.net/azureml/ExperimentRun/dcid.ba4d85cf-d82e-4c7c-bb78-325838b37271/logs/azureml/executionlogs.txt?sv=2019-07-07&sr=b&sig=YA2yGF0HGJ2ZLePiOKmt0ouV56VU0q6Zh0Z1%2FXZU6iw%3D&st=2021-08-27T08%3A35%3A58Z&se=2021-08-27T16%3A45%3A58Z&sp=r', 'logs/azureml/job_prep_azureml.log': 'https://azuremlworksho7897742112.blob.core.windows.net/azureml/ExperimentRun/dcid.ba4d85cf-d82e-4c7c-bb78-325838b37271/logs/azureml/job_prep_azureml.log?sv=2019-07-07&sr=b&sig=%2FM9w6ROHGeblU4Sv4pwnxnoWlWBcpqjB0VQ1rJNIzMQ%3D&st=2021-08-27T08%3A35%3A58Z&se=2021-08-27T16%3A45%3A58Z&sp=r', 'logs/azureml/job_release_azureml.log': 'https://azuremlworksho7897742112.blob.core.windows.net/azureml/ExperimentRun/dcid.ba4d85cf-d82e-4c7c-bb78-325838b37271/logs/azureml/job_release_azureml.log?sv=2019-07-07&sr=b&sig=1U%2FksdLQrKT0RIHapBabL3ZBu3cYrCUgImGbIFVHyBA%3D&st=2021-08-27T08%3A35%3A58Z&se=2021-08-27T16%3A45%3A58Z&sp=r', 'logs/azureml/sidecar/tvmps_7f60c2ab84f5ed82a91597097db1e256bcdf0668aa1fbdf25cbbb7c0d9323244_d/all.log': 'https://azuremlworksho7897742112.blob.core.windows.net/azureml/ExperimentRun/dcid.ba4d85cf-d82e-4c7c-bb78-325838b37271/logs/azureml/sidecar/tvmps_7f60c2ab84f5ed82a91597097db1e256bcdf0668aa1fbdf25cbbb7c0d9323244_d/all.log?sv=2019-07-07&sr=b&sig=toqg0nO5AEBnE6TwLRc0a79IC%2B3hzXsRBwY2rqiGD14%3D&st=2021-08-27T08%3A35%3A58Z&se=2021-08-27T16%3A45%3A58Z&sp=r', 'logs/azureml/sidecar/tvmps_7f60c2ab84f5ed82a91597097db1e256bcdf0668aa1fbdf25cbbb7c0d9323244_d/task.enter_contexts.log': 'https://azuremlworksho7897742112.blob.core.windows.net/azureml/ExperimentRun/dcid.ba4d85cf-d82e-4c7c-bb78-325838b37271/logs/azureml/sidecar/tvmps_7f60c2ab84f5ed82a91597097db1e256bcdf0668aa1fbdf25cbbb7c0d9323244_d/task.enter_contexts.log?sv=2019-07-07&sr=b&sig=ylAtZBKMq5irPdlfBXbsmqXVZhk0BiAGmX2cHzgfQlE%3D&st=2021-08-27T08%3A35%3A58Z&se=2021-08-27T16%3A45%3A58Z&sp=r', 'logs/azureml/sidecar/tvmps_7f60c2ab84f5ed82a91597097db1e256bcdf0668aa1fbdf25cbbb7c0d9323244_d/task.exit_contexts.log': 'https://azuremlworksho7897742112.blob.core.windows.net/azureml/ExperimentRun/dcid.ba4d85cf-d82e-4c7c-bb78-325838b37271/logs/azureml/sidecar/tvmps_7f60c2ab84f5ed82a91597097db1e256bcdf0668aa1fbdf25cbbb7c0d9323244_d/task.exit_contexts.log?sv=2019-07-07&sr=b&sig=f%2BDcBd6c2NwW4pgHy2BrzfzqgltIql7FlRvSE6nJRZo%3D&st=2021-08-27T08%3A35%3A58Z&se=2021-08-27T16%3A45%3A58Z&sp=r', 'logs/azureml/sidecar/tvmps_f7393ee5d1f97ad75e43509d348b2677c577350c27484eaa7b0d5ef01cbfa2e6_d/all.log': 'https://azuremlworksho7897742112.blob.core.windows.net/azureml/ExperimentRun/dcid.ba4d85cf-d82e-4c7c-bb78-325838b37271/logs/azureml/sidecar/tvmps_f7393ee5d1f97ad75e43509d348b2677c577350c27484eaa7b0d5ef01cbfa2e6_d/all.log?sv=2019-07-07&sr=b&sig=X1Xl9SjjbW24kdF2SwUtQ5xzSrV1qKhOJVADDyuMUQw%3D&st=2021-08-27T08%3A35%3A58Z&se=2021-08-27T16%3A45%3A58Z&sp=r', 'logs/azureml/sidecar/tvmps_f7393ee5d1f97ad75e43509d348b2677c577350c27484eaa7b0d5ef01cbfa2e6_d/task.enter_contexts.log': 'https://azuremlworksho7897742112.blob.core.windows.net/azureml/ExperimentRun/dcid.ba4d85cf-d82e-4c7c-bb78-325838b37271/logs/azureml/sidecar/tvmps_f7393ee5d1f97ad75e43509d348b2677c577350c27484eaa7b0d5ef01cbfa2e6_d/task.enter_contexts.log?sv=2019-07-07&sr=b&sig=TeY88QoROPd8FiJf7%2B2I9Vcgo%2FKeHiw1SdFDTK2jirk%3D&st=2021-08-27T08%3A35%3A58Z&se=2021-08-27T16%3A45%3A58Z&sp=r', 'logs/azureml/sidecar/tvmps_f7393ee5d1f97ad75e43509d348b2677c577350c27484eaa7b0d5ef01cbfa2e6_d/task.exit_contexts.log': 'https://azuremlworksho7897742112.blob.core.windows.net/azureml/ExperimentRun/dcid.ba4d85cf-d82e-4c7c-bb78-325838b37271/logs/azureml/sidecar/tvmps_f7393ee5d1f97ad75e43509d348b2677c577350c27484eaa7b0d5ef01cbfa2e6_d/task.exit_contexts.log?sv=2019-07-07&sr=b&sig=il7ag5u%2BljhbXkekDQAHJAYwlcbOanbo28nTyIhxsKg%3D&st=2021-08-27T08%3A35%3A58Z&se=2021-08-27T16%3A45%3A58Z&sp=r', 'logs/azureml/stderrlogs.txt': 'https://azuremlworksho7897742112.blob.core.windows.net/azureml/ExperimentRun/dcid.ba4d85cf-d82e-4c7c-bb78-325838b37271/logs/azureml/stderrlogs.txt?sv=2019-07-07&sr=b&sig=KJnxxax8yNjAmC3WR1N8EQhpmE3sTVHHPHGQioiU9es%3D&st=2021-08-27T08%3A35%3A58Z&se=2021-08-27T16%3A45%3A58Z&sp=r', 'logs/azureml/stdoutlogs.txt': 'https://azuremlworksho7897742112.blob.core.windows.net/azureml/ExperimentRun/dcid.ba4d85cf-d82e-4c7c-bb78-325838b37271/logs/azureml/stdoutlogs.txt?sv=2019-07-07&sr=b&sig=jxFnY4rTA41JZtlivvyuXYzOlGUEl2w7akIfOEL5ZLw%3D&st=2021-08-27T08%3A35%3A58Z&se=2021-08-27T16%3A45%3A58Z&sp=r'}, 'submittedBy': 'Serge Retkowsky'}\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "PipelineRun Execution Summary\n",
      "==============================\n",
      "PipelineRun Status: Finished\n",
      "{'runId': 'ba67ad10-2e7b-451f-bf30-f957e21cf12b', 'status': 'Completed', 'startTimeUtc': '2021-08-27T08:43:30.601332Z', 'endTimeUtc': '2021-08-27T08:46:13.998114Z', 'properties': {'azureml.runsource': 'azureml.PipelineRun', 'runSource': 'SDK', 'runType': 'SDK', 'azureml.parameters': '{\"batch_size_param\":\"1\",\"process_count_param\":\"1\"}'}, 'inputDatasets': [], 'outputDatasets': [], 'logFiles': {'logs/azureml/executionlogs.txt': 'https://azuremlworksho7897742112.blob.core.windows.net/azureml/ExperimentRun/dcid.ba67ad10-2e7b-451f-bf30-f957e21cf12b/logs/azureml/executionlogs.txt?sv=2019-07-07&sr=b&sig=QyDhfHGvCotc0XVR0pcXDeKv3zs%2Bed5%2B%2FDNuBLQuouA%3D&st=2021-08-27T08%3A36%3A15Z&se=2021-08-27T16%3A46%3A15Z&sp=r', 'logs/azureml/stderrlogs.txt': 'https://azuremlworksho7897742112.blob.core.windows.net/azureml/ExperimentRun/dcid.ba67ad10-2e7b-451f-bf30-f957e21cf12b/logs/azureml/stderrlogs.txt?sv=2019-07-07&sr=b&sig=AHQi7KJIFbixe%2FzjUI%2BZB5aoegioPmrMzksyx0SMso4%3D&st=2021-08-27T08%3A36%3A15Z&se=2021-08-27T16%3A46%3A15Z&sp=r', 'logs/azureml/stdoutlogs.txt': 'https://azuremlworksho7897742112.blob.core.windows.net/azureml/ExperimentRun/dcid.ba67ad10-2e7b-451f-bf30-f957e21cf12b/logs/azureml/stdoutlogs.txt?sv=2019-07-07&sr=b&sig=%2FzNRmj6d6OBT5tdbB4FwxPfI96mKD5QwMcf23V8Weaw%3D&st=2021-08-27T08%3A36%3A15Z&se=2021-08-27T16%3A46%3A15Z&sp=r'}, 'submittedBy': 'Serge Retkowsky'}\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'Finished'"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Wait the run for completion and show output log to console\n",
    "pipeline_run_2.wait_for_completion(show_output=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The pipeline is Finished at the moment ( 2021-08-27 08:46:15.612368 )\n"
     ]
    }
   ],
   "source": [
    "print(\"The pipeline is\", pipeline_run_2.get_status(), 'at the moment (', datetime.datetime.now(), ')')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Cleanup Compute resources\n",
    "\n",
    "For re-occurring jobs, it may be wise to keep compute the compute resources and allow compute nodes to scale down to 0. However, since this is just a single-run job, we are free to release the allocated compute resources."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "compute_target.delete()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## End of labs"
   ]
  }
 ],
 "metadata": {
  "authors": [
   {
    "name": "joringer"
   },
   {
    "name": "asraniwa"
   },
   {
    "name": "pansav"
   },
   {
    "name": "tracych"
   }
  ],
  "category": "Other notebooks",
  "compute": [
   "AML Compute"
  ],
  "datasets": [
   "MNIST"
  ],
  "deployment": [
   "None"
  ],
  "exclude_from_index": false,
  "framework": [
   "None"
  ],
  "friendly_name": "MNIST data inferencing using ParallelRunStep",
  "index_order": 1,
  "kernelspec": {
   "display_name": "Python 3.6 - AzureML",
   "language": "python",
   "name": "python3-azureml"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  },
  "tags": [
   "Batch Inferencing",
   "Pipeline"
  ],
  "task": "Digit identification"
 },
 "nbformat": 4,
 "nbformat_minor": 2
}