5 AML Compute.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Azure ML Compute"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='https://github.com/retkowsky/images/blob/master/AzureMLservicebanniere.png?raw=true'>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Documentation:<br>\n",
    "https://docs.microsoft.com/en-us/azure/machine-learning/concept-compute-target <br>\n",
    "https://docs.microsoft.com/en-us/azure/machine-learning/how-to-set-up-training-targets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "**Azure Machine Learning Compute** is a **managed-compute infrastructure** that allows the user to easily create a single or multi-node compute. The compute is created within your workspace region as a resource that can be shared with other users in your workspace. The compute **scales up automatically when a job is submitted**, and can be put in an Azure Virtual Network. The compute executes in a containerized environment and packages your model dependencies in a **Docker container**.\n",
    "\n",
    "You can use Azure Machine Learning Compute to distribute the training process across a cluster of **CPU or GPU** compute nodes in the cloud. For more information on the VM sizes that include GPUs, see GPU-optimized virtual machine sizes.\n",
    "\n",
    "Azure Machine Learning Compute has default limits, such as the number of cores that can be allocated. For more information, see Manage and request quotas for Azure resources.\n",
    "\n",
    "You can create an Azure Machine Learning compute environment **on demand** when you schedule a run, or as a **persistent resource**.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Intro"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'3.6.9 |Anaconda, Inc.| (default, Jul 30 2019, 19:07:31) \\n[GCC 7.3.0]'"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import sys\n",
    "sys.version"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2020-09-16 12:46:22.970519\n"
     ]
    }
   ],
   "source": [
    "import datetime\n",
    "now = datetime.datetime.now()\n",
    "print(now)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Version Azure ML : 1.13.0\n"
     ]
    }
   ],
   "source": [
    "import azureml.core\n",
    "print(\"Version Azure ML :\", azureml.core.VERSION)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Workspace\n",
    "\n",
    "Initialize a workspace object from persisted configuration"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "tags": [
     "create workspace"
    ]
   },
   "outputs": [],
   "source": [
    "from azureml.core import Workspace\n",
    "\n",
    "ws = Workspace.from_config()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Expérimentation\n",
    "\n",
    "**Experiment** is a logical container in an Azure ML Workspace. It hosts run records which can include run metrics and output artifacts from your experiments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.core import Experiment\n",
    "experiment_name = 'Exemple5-amlcompute'\n",
    "experiment = Experiment(workspace = ws, name = experiment_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Introduction AmlCompute"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-set-up-training-targets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Liste des compute servers définis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Compute instances in your workspace Azure ML:\n",
      "- AutoML\n",
      "- Designer\n",
      "- notebooksjupyter\n"
     ]
    }
   ],
   "source": [
    "print(\"Compute instances in your workspace Azure ML:\")\n",
    "cts = ws.compute_targets\n",
    "for ct in cts:\n",
    "    print('-', ct)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.1 Liste serveurs AML Compute disponibles"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'name': 'Standard_D1_v2',\n",
       "  'vCPUs': 1,\n",
       "  'gpus': 0,\n",
       "  'memoryGB': 3.5,\n",
       "  'maxResourceVolumeMB': 51200},\n",
       " {'name': 'Standard_D2_v2',\n",
       "  'vCPUs': 2,\n",
       "  'gpus': 0,\n",
       "  'memoryGB': 7.0,\n",
       "  'maxResourceVolumeMB': 102400},\n",
       " {'name': 'Standard_D3_v2',\n",
       "  'vCPUs': 4,\n",
       "  'gpus': 0,\n",
       "  'memoryGB': 14.0,\n",
       "  'maxResourceVolumeMB': 204800},\n",
       " {'name': 'Standard_D4_v2',\n",
       "  'vCPUs': 8,\n",
       "  'gpus': 0,\n",
       "  'memoryGB': 28.0,\n",
       "  'maxResourceVolumeMB': 409600},\n",
       " {'name': 'Standard_D11_v2',\n",
       "  'vCPUs': 2,\n",
       "  'gpus': 0,\n",
       "  'memoryGB': 14.0,\n",
       "  'maxResourceVolumeMB': 102400},\n",
       " {'name': 'Standard_D12_v2',\n",
       "  'vCPUs': 4,\n",
       "  'gpus': 0,\n",
       "  'memoryGB': 28.0,\n",
       "  'maxResourceVolumeMB': 204800},\n",
       " {'name': 'Standard_D13_v2',\n",
       "  'vCPUs': 8,\n",
       "  'gpus': 0,\n",
       "  'memoryGB': 56.0,\n",
       "  'maxResourceVolumeMB': 409600},\n",
       " {'name': 'Standard_D14_v2',\n",
       "  'vCPUs': 16,\n",
       "  'gpus': 0,\n",
       "  'memoryGB': 112.0,\n",
       "  'maxResourceVolumeMB': 819200},\n",
       " {'name': 'Standard_DS1_v2',\n",
       "  'vCPUs': 1,\n",
       "  'gpus': 0,\n",
       "  'memoryGB': 3.5,\n",
       "  'maxResourceVolumeMB': 7168},\n",
       " {'name': 'Standard_DS2_v2',\n",
       "  'vCPUs': 2,\n",
       "  'gpus': 0,\n",
       "  'memoryGB': 7.0,\n",
       "  'maxResourceVolumeMB': 14336},\n",
       " {'name': 'Standard_DS3_v2',\n",
       "  'vCPUs': 4,\n",
       "  'gpus': 0,\n",
       "  'memoryGB': 14.0,\n",
       "  'maxResourceVolumeMB': 28672},\n",
       " {'name': 'Standard_DS4_v2',\n",
       "  'vCPUs': 8,\n",
       "  'gpus': 0,\n",
       "  'memoryGB': 28.0,\n",
       "  'maxResourceVolumeMB': 57344},\n",
       " {'name': 'Standard_DS5_v2',\n",
       "  'vCPUs': 16,\n",
       "  'gpus': 0,\n",
       "  'memoryGB': 56.0,\n",
       "  'maxResourceVolumeMB': 114688},\n",
       " {'name': 'Standard_DS11_v2',\n",
       "  'vCPUs': 2,\n",
       "  'gpus': 0,\n",
       "  'memoryGB': 14.0,\n",
       "  'maxResourceVolumeMB': 28672},\n",
       " {'name': 'Standard_DS12_v2',\n",
       "  'vCPUs': 4,\n",
       "  'gpus': 0,\n",
       "  'memoryGB': 28.0,\n",
       "  'maxResourceVolumeMB': 57344},\n",
       " {'name': 'Standard_DS13_v2',\n",
       "  'vCPUs': 8,\n",
       "  'gpus': 0,\n",
       "  'memoryGB': 56.0,\n",
       "  'maxResourceVolumeMB': 114688},\n",
       " {'name': 'Standard_DS14_v2',\n",
       "  'vCPUs': 16,\n",
       "  'gpus': 0,\n",
       "  'memoryGB': 112.0,\n",
       "  'maxResourceVolumeMB': 229376},\n",
       " {'name': 'Standard_D2_v3',\n",
       "  'vCPUs': 2,\n",
       "  'gpus': 0,\n",
       "  'memoryGB': 8.0,\n",
       "  'maxResourceVolumeMB': 51200},\n",
       " {'name': 'Standard_D4_v3',\n",
       "  'vCPUs': 4,\n",
       "  'gpus': 0,\n",
       "  'memoryGB': 16.0,\n",
       "  'maxResourceVolumeMB': 102400},\n",
       " {'name': 'Standard_D8_v3',\n",
       "  'vCPUs': 8,\n",
       "  'gpus': 0,\n",
       "  'memoryGB': 32.0,\n",
       "  'maxResourceVolumeMB': 204800},\n",
       " {'name': 'Standard_D16_v3',\n",
       "  'vCPUs': 16,\n",
       "  'gpus': 0,\n",
       "  'memoryGB': 64.0,\n",
       "  'maxResourceVolumeMB': 409600},\n",
       " {'name': 'Standard_D32_v3',\n",
       "  'vCPUs': 32,\n",
       "  'gpus': 0,\n",
       "  'memoryGB': 128.0,\n",
       "  'maxResourceVolumeMB': 819200},\n",
       " {'name': 'Standard_D64_v3',\n",
       "  'vCPUs': 64,\n",
       "  'gpus': 0,\n",
       "  'memoryGB': 256.0,\n",
       "  'maxResourceVolumeMB': 1638400},\n",
       " {'name': 'Standard_D2s_v3',\n",
       "  'vCPUs': 2,\n",
       "  'gpus': 0,\n",
       "  'memoryGB': 8.0,\n",
       "  'maxResourceVolumeMB': 16384},\n",
       " {'name': 'Standard_D4s_v3',\n",
       "  'vCPUs': 4,\n",
       "  'gpus': 0,\n",
       "  'memoryGB': 16.0,\n",
       "  'maxResourceVolumeMB': 32768},\n",
       " {'name': 'Standard_D8s_v3',\n",
       "  'vCPUs': 8,\n",
       "  'gpus': 0,\n",
       "  'memoryGB': 32.0,\n",
       "  'maxResourceVolumeMB': 65536},\n",
       " {'name': 'Standard_D16s_v3',\n",
       "  'vCPUs': 16,\n",
       "  'gpus': 0,\n",
       "  'memoryGB': 64.0,\n",
       "  'maxResourceVolumeMB': 131072},\n",
       " {'name': 'Standard_D32s_v3',\n",
       "  'vCPUs': 32,\n",
       "  'gpus': 0,\n",
       "  'memoryGB': 128.0,\n",
       "  'maxResourceVolumeMB': 262144},\n",
       " {'name': 'Standard_D64s_v3',\n",
       "  'vCPUs': 64,\n",
       "  'gpus': 0,\n",
       "  'memoryGB': 256.0,\n",
       "  'maxResourceVolumeMB': 524288},\n",
       " {'name': 'Standard_M8-2ms',\n",
       "  'vCPUs': 8,\n",
       "  'gpus': 0,\n",
       "  'memoryGB': 218.75,\n",
       "  'maxResourceVolumeMB': 256000},\n",
       " {'name': 'Standard_M8-4ms',\n",
       "  'vCPUs': 8,\n",
       "  'gpus': 0,\n",
       "  'memoryGB': 218.75,\n",
       "  'maxResourceVolumeMB': 256000},\n",
       " {'name': 'Standard_M8ms',\n",
       "  'vCPUs': 8,\n",
       "  'gpus': 0,\n",
       "  'memoryGB': 218.75,\n",
       "  'maxResourceVolumeMB': 256000},\n",
       " {'name': 'Standard_M16-4ms',\n",
       "  'vCPUs': 16,\n",
       "  'gpus': 0,\n",
       "  'memoryGB': 437.5,\n",
       "  'maxResourceVolumeMB': 512000},\n",
       " {'name': 'Standard_M16-8ms',\n",
       "  'vCPUs': 16,\n",
       "  'gpus': 0,\n",
       "  'memoryGB': 437.5,\n",
       "  'maxResourceVolumeMB': 512000},\n",
       " {'name': 'Standard_M16ms',\n",
       "  'vCPUs': 16,\n",
       "  'gpus': 0,\n",
       "  'memoryGB': 437.5,\n",
       "  'maxResourceVolumeMB': 512000},\n",
       " {'name': 'Standard_M32-8ms',\n",
       "  'vCPUs': 32,\n",
       "  'gpus': 0,\n",
       "  'memoryGB': 875.0,\n",
       "  'maxResourceVolumeMB': 1024000},\n",
       " {'name': 'Standard_M32-16ms',\n",
       "  'vCPUs': 32,\n",
       "  'gpus': 0,\n",
       "  'memoryGB': 875.0,\n",
       "  'maxResourceVolumeMB': 1024000},\n",
       " {'name': 'Standard_M32ls',\n",
       "  'vCPUs': 32,\n",
       "  'gpus': 0,\n",
       "  'memoryGB': 256.0,\n",
       "  'maxResourceVolumeMB': 1024000},\n",
       " {'name': 'Standard_M32ms',\n",
       "  'vCPUs': 32,\n",
       "  'gpus': 0,\n",
       "  'memoryGB': 875.0,\n",
       "  'maxResourceVolumeMB': 1024000},\n",
       " {'name': 'Standard_M32ts',\n",
       "  'vCPUs': 32,\n",
       "  'gpus': 0,\n",
       "  'memoryGB': 192.0,\n",
       "  'maxResourceVolumeMB': 1024000},\n",
       " {'name': 'Standard_M64-16ms',\n",
       "  'vCPUs': 64,\n",
       "  'gpus': 0,\n",
       "  'memoryGB': 1750.0,\n",
       "  'maxResourceVolumeMB': 2048000},\n",
       " {'name': 'Standard_M64-32ms',\n",
       "  'vCPUs': 64,\n",
       "  'gpus': 0,\n",
       "  'memoryGB': 1750.0,\n",
       "  'maxResourceVolumeMB': 2048000},\n",
       " {'name': 'Standard_M64ls',\n",
       "  'vCPUs': 64,\n",
       "  'gpus': 0,\n",
       "  'memoryGB': 512.0,\n",
       "  'maxResourceVolumeMB': 2048000},\n",
       " {'name': 'Standard_M64ms',\n",
       "  'vCPUs': 64,\n",
       "  'gpus': 0,\n",
       "  'memoryGB': 1750.0,\n",
       "  'maxResourceVolumeMB': 2048000},\n",
       " {'name': 'Standard_M64s',\n",
       "  'vCPUs': 64,\n",
       "  'gpus': 0,\n",
       "  'memoryGB': 1000.0,\n",
       "  'maxResourceVolumeMB': 2048000},\n",
       " {'name': 'Standard_M128-32ms',\n",
       "  'vCPUs': 128,\n",
       "  'gpus': 0,\n",
       "  'memoryGB': 3800.0,\n",
       "  'maxResourceVolumeMB': 4096000},\n",
       " {'name': 'Standard_M128-64ms',\n",
       "  'vCPUs': 128,\n",
       "  'gpus': 0,\n",
       "  'memoryGB': 3800.0,\n",
       "  'maxResourceVolumeMB': 4096000},\n",
       " {'name': 'Standard_M128ms',\n",
       "  'vCPUs': 128,\n",
       "  'gpus': 0,\n",
       "  'memoryGB': 3800.0,\n",
       "  'maxResourceVolumeMB': 4096000},\n",
       " {'name': 'Standard_M128s',\n",
       "  'vCPUs': 128,\n",
       "  'gpus': 0,\n",
       "  'memoryGB': 2000.0,\n",
       "  'maxResourceVolumeMB': 4096000},\n",
       " {'name': 'Standard_M64',\n",
       "  'vCPUs': 64,\n",
       "  'gpus': 0,\n",
       "  'memoryGB': 1000.0,\n",
       "  'maxResourceVolumeMB': 8192000},\n",
       " {'name': 'Standard_M64m',\n",
       "  'vCPUs': 64,\n",
       "  'gpus': 0,\n",
       "  'memoryGB': 1750.0,\n",
       "  'maxResourceVolumeMB': 8192000},\n",
       " {'name': 'Standard_M128',\n",
       "  'vCPUs': 128,\n",
       "  'gpus': 0,\n",
       "  'memoryGB': 2000.0,\n",
       "  'maxResourceVolumeMB': 16384000},\n",
       " {'name': 'Standard_M128m',\n",
       "  'vCPUs': 128,\n",
       "  'gpus': 0,\n",
       "  'memoryGB': 3800.0,\n",
       "  'maxResourceVolumeMB': 16384000},\n",
       " {'name': 'Standard_D1',\n",
       "  'vCPUs': 1,\n",
       "  'gpus': 0,\n",
       "  'memoryGB': 3.5,\n",
       "  'maxResourceVolumeMB': 51200},\n",
       " {'name': 'Standard_D2',\n",
       "  'vCPUs': 2,\n",
       "  'gpus': 0,\n",
       "  'memoryGB': 7.0,\n",
       "  'maxResourceVolumeMB': 102400},\n",
       " {'name': 'Standard_D3',\n",
       "  'vCPUs': 4,\n",
       "  'gpus': 0,\n",
       "  'memoryGB': 14.0,\n",
       "  'maxResourceVolumeMB': 204800},\n",
       " {'name': 'Standard_D4',\n",
       "  'vCPUs': 8,\n",
       "  'gpus': 0,\n",
       "  'memoryGB': 28.0,\n",
       "  'maxResourceVolumeMB': 409600},\n",
       " {'name': 'Standard_D11',\n",
       "  'vCPUs': 2,\n",
       "  'gpus': 0,\n",
       "  'memoryGB': 14.0,\n",
       "  'maxResourceVolumeMB': 102400},\n",
       " {'name': 'Standard_D12',\n",
       "  'vCPUs': 4,\n",
       "  'gpus': 0,\n",
       "  'memoryGB': 28.0,\n",
       "  'maxResourceVolumeMB': 204800},\n",
       " {'name': 'Standard_D13',\n",
       "  'vCPUs': 8,\n",
       "  'gpus': 0,\n",
       "  'memoryGB': 56.0,\n",
       "  'maxResourceVolumeMB': 409600},\n",
       " {'name': 'Standard_D14',\n",
       "  'vCPUs': 16,\n",
       "  'gpus': 0,\n",
       "  'memoryGB': 112.0,\n",
       "  'maxResourceVolumeMB': 819200},\n",
       " {'name': 'Standard_HB120rs_v2',\n",
       "  'vCPUs': 120,\n",
       "  'gpus': 0,\n",
       "  'memoryGB': 468.75,\n",
       "  'maxResourceVolumeMB': 960000},\n",
       " {'name': 'Standard_D15_v2',\n",
       "  'vCPUs': 20,\n",
       "  'gpus': 0,\n",
       "  'memoryGB': 140.0,\n",
       "  'maxResourceVolumeMB': 1024000},\n",
       " {'name': 'Standard_DS15_v2',\n",
       "  'vCPUs': 20,\n",
       "  'gpus': 0,\n",
       "  'memoryGB': 140.0,\n",
       "  'maxResourceVolumeMB': 286720},\n",
       " {'name': 'Standard_F2s_v2',\n",
       "  'vCPUs': 2,\n",
       "  'gpus': 0,\n",
       "  'memoryGB': 4.0,\n",
       "  'maxResourceVolumeMB': 16384},\n",
       " {'name': 'Standard_F4s_v2',\n",
       "  'vCPUs': 4,\n",
       "  'gpus': 0,\n",
       "  'memoryGB': 8.0,\n",
       "  'maxResourceVolumeMB': 32768},\n",
       " {'name': 'Standard_F8s_v2',\n",
       "  'vCPUs': 8,\n",
       "  'gpus': 0,\n",
       "  'memoryGB': 16.0,\n",
       "  'maxResourceVolumeMB': 65536},\n",
       " {'name': 'Standard_F16s_v2',\n",
       "  'vCPUs': 16,\n",
       "  'gpus': 0,\n",
       "  'memoryGB': 32.0,\n",
       "  'maxResourceVolumeMB': 131072},\n",
       " {'name': 'Standard_F32s_v2',\n",
       "  'vCPUs': 32,\n",
       "  'gpus': 0,\n",
       "  'memoryGB': 64.0,\n",
       "  'maxResourceVolumeMB': 262144},\n",
       " {'name': 'Standard_F64s_v2',\n",
       "  'vCPUs': 64,\n",
       "  'gpus': 0,\n",
       "  'memoryGB': 128.0,\n",
       "  'maxResourceVolumeMB': 524288},\n",
       " {'name': 'Standard_F72s_v2',\n",
       "  'vCPUs': 72,\n",
       "  'gpus': 0,\n",
       "  'memoryGB': 144.0,\n",
       "  'maxResourceVolumeMB': 589824},\n",
       " {'name': 'Standard_NV6',\n",
       "  'vCPUs': 6,\n",
       "  'gpus': 1,\n",
       "  'memoryGB': 56.0,\n",
       "  'maxResourceVolumeMB': 389120},\n",
       " {'name': 'Standard_NV12',\n",
       "  'vCPUs': 12,\n",
       "  'gpus': 2,\n",
       "  'memoryGB': 112.0,\n",
       "  'maxResourceVolumeMB': 696320},\n",
       " {'name': 'Standard_NV24',\n",
       "  'vCPUs': 24,\n",
       "  'gpus': 4,\n",
       "  'memoryGB': 224.0,\n",
       "  'maxResourceVolumeMB': 1474560},\n",
       " {'name': 'Standard_NV12s_v3',\n",
       "  'vCPUs': 12,\n",
       "  'gpus': 1,\n",
       "  'memoryGB': 112.0,\n",
       "  'maxResourceVolumeMB': 344064},\n",
       " {'name': 'Standard_NV24s_v3',\n",
       "  'vCPUs': 24,\n",
       "  'gpus': 2,\n",
       "  'memoryGB': 224.0,\n",
       "  'maxResourceVolumeMB': 688128},\n",
       " {'name': 'Standard_NV48s_v3',\n",
       "  'vCPUs': 48,\n",
       "  'gpus': 4,\n",
       "  'memoryGB': 448.0,\n",
       "  'maxResourceVolumeMB': 1376256},\n",
       " {'name': 'Standard_NC6s_v3',\n",
       "  'vCPUs': 6,\n",
       "  'gpus': 1,\n",
       "  'memoryGB': 112.0,\n",
       "  'maxResourceVolumeMB': 344064},\n",
       " {'name': 'Standard_NC12s_v3',\n",
       "  'vCPUs': 12,\n",
       "  'gpus': 2,\n",
       "  'memoryGB': 224.0,\n",
       "  'maxResourceVolumeMB': 688128},\n",
       " {'name': 'Standard_NC24rs_v3',\n",
       "  'vCPUs': 24,\n",
       "  'gpus': 4,\n",
       "  'memoryGB': 448.0,\n",
       "  'maxResourceVolumeMB': 1376256},\n",
       " {'name': 'Standard_NC24s_v3',\n",
       "  'vCPUs': 24,\n",
       "  'gpus': 4,\n",
       "  'memoryGB': 448.0,\n",
       "  'maxResourceVolumeMB': 1376256},\n",
       " {'name': 'Standard_NC6',\n",
       "  'vCPUs': 6,\n",
       "  'gpus': 1,\n",
       "  'memoryGB': 56.0,\n",
       "  'maxResourceVolumeMB': 389120},\n",
       " {'name': 'Standard_NC12',\n",
       "  'vCPUs': 12,\n",
       "  'gpus': 2,\n",
       "  'memoryGB': 112.0,\n",
       "  'maxResourceVolumeMB': 696320},\n",
       " {'name': 'Standard_NC24',\n",
       "  'vCPUs': 24,\n",
       "  'gpus': 4,\n",
       "  'memoryGB': 224.0,\n",
       "  'maxResourceVolumeMB': 1474560},\n",
       " {'name': 'Standard_NC24r',\n",
       "  'vCPUs': 24,\n",
       "  'gpus': 4,\n",
       "  'memoryGB': 224.0,\n",
       "  'maxResourceVolumeMB': 1474560},\n",
       " {'name': 'Standard_ND6s',\n",
       "  'vCPUs': 6,\n",
       "  'gpus': 1,\n",
       "  'memoryGB': 112.0,\n",
       "  'maxResourceVolumeMB': 344064},\n",
       " {'name': 'Standard_ND12s',\n",
       "  'vCPUs': 12,\n",
       "  'gpus': 2,\n",
       "  'memoryGB': 224.0,\n",
       "  'maxResourceVolumeMB': 688128},\n",
       " {'name': 'Standard_ND24rs',\n",
       "  'vCPUs': 24,\n",
       "  'gpus': 4,\n",
       "  'memoryGB': 448.0,\n",
       "  'maxResourceVolumeMB': 1376256},\n",
       " {'name': 'Standard_ND24s',\n",
       "  'vCPUs': 24,\n",
       "  'gpus': 4,\n",
       "  'memoryGB': 448.0,\n",
       "  'maxResourceVolumeMB': 1376256},\n",
       " {'name': 'Standard_ND40rs_v2',\n",
       "  'vCPUs': 40,\n",
       "  'gpus': 8,\n",
       "  'memoryGB': 672.0,\n",
       "  'maxResourceVolumeMB': 2969600},\n",
       " {'name': 'Standard_HC44rs',\n",
       "  'vCPUs': 44,\n",
       "  'gpus': 0,\n",
       "  'memoryGB': 327.83,\n",
       "  'maxResourceVolumeMB': 716800},\n",
       " {'name': 'Standard_NC6s_v2',\n",
       "  'vCPUs': 6,\n",
       "  'gpus': 1,\n",
       "  'memoryGB': 112.0,\n",
       "  'maxResourceVolumeMB': 344064},\n",
       " {'name': 'Standard_NC12s_v2',\n",
       "  'vCPUs': 12,\n",
       "  'gpus': 2,\n",
       "  'memoryGB': 224.0,\n",
       "  'maxResourceVolumeMB': 688128},\n",
       " {'name': 'Standard_NC24rs_v2',\n",
       "  'vCPUs': 24,\n",
       "  'gpus': 4,\n",
       "  'memoryGB': 448.0,\n",
       "  'maxResourceVolumeMB': 1376256},\n",
       " {'name': 'Standard_NC24s_v2',\n",
       "  'vCPUs': 24,\n",
       "  'gpus': 4,\n",
       "  'memoryGB': 448.0,\n",
       "  'maxResourceVolumeMB': 1376256}]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from azureml.core.compute import ComputeTarget, AmlCompute\n",
    "\n",
    "AmlCompute.supported_vmsizes(workspace = ws)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.2 Répertoire\n",
    "\n",
    "Create a directory that will contain all the necessary code from your local machine that you will need access to on the remote resource. This includes the training script, and any additional files your training script depends on"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'./train-on-amlcompute/train_aml.py'"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import os\n",
    "import shutil\n",
    "\n",
    "project_folder = './train-on-amlcompute'\n",
    "os.makedirs(project_folder, exist_ok=True)\n",
    "shutil.copy('train_aml.py', project_folder)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "# Copyright (c) Microsoft. All rights reserved.\n",
      "# Licensed under the MIT license.\n",
      "\n",
      "from sklearn.datasets import load_diabetes\n",
      "from sklearn.linear_model import Ridge\n",
      "from sklearn.metrics import mean_squared_error\n",
      "from sklearn.model_selection import train_test_split\n",
      "from azureml.core.run import Run\n",
      "from sklearn.externals import joblib\n",
      "import os\n",
      "import numpy as np\n",
      "\n",
      "os.makedirs('./outputs', exist_ok=True)\n",
      "\n",
      "X, y = load_diabetes(return_X_y=True)\n",
      "\n",
      "run = Run.get_context()\n",
      "\n",
      "X_train, X_test, y_train, y_test = train_test_split(X, y,\n",
      "                                                    test_size=0.2,\n",
      "                                                    random_state=0)\n",
      "data = {\"train\": {\"X\": X_train, \"y\": y_train},\n",
      "        \"test\": {\"X\": X_test, \"y\": y_test}}\n",
      "\n",
      "# list of numbers from 0.0 to 1.0 with a 0.05 interval\n",
      "alphas = np.arange(0.0, 1.0, 0.05)\n",
      "\n",
      "for alpha in alphas:\n",
      "    # Use Ridge algorithm to create a regression model\n",
      "    reg = Ridge(alpha=alpha)\n",
      "    reg.fit(data[\"train\"][\"X\"], data[\"train\"][\"y\"])\n",
      "\n",
      "    preds = reg.predict(data[\"test\"][\"X\"])\n",
      "    mse = mean_squared_error(preds, data[\"test\"][\"y\"])\n",
      "    run.log('alpha', alpha)\n",
      "    run.log('mse', mse)\n",
      "\n",
      "    model_file_name = 'ridge_{0:.2f}.pkl'.format(alpha)\n",
      "    # save model in the outputs folder so it automatically get uploaded\n",
      "    with open(model_file_name, \"wb\") as file:\n",
      "        joblib.dump(value=reg, filename=os.path.join('./outputs/',\n",
      "                                                     model_file_name))\n",
      "\n",
      "    print('alpha is {0:.2f}, and mse is {1:0.2f}'.format(alpha, mse))\n",
      "\n"
     ]
    }
   ],
   "source": [
    "with open(os.path.join('./train-on-amlcompute/train_aml.py'), 'r') as f:\n",
    "    print(f.read())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Version scikit-learn = 0.20.3\n"
     ]
    }
   ],
   "source": [
    "import sklearn\n",
    "print('Version scikit-learn =', sklearn.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.3 Environnement"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.core import Environment\n",
    "from azureml.core.conda_dependencies import CondaDependencies\n",
    "\n",
    "myenv = Environment(\"myenv\")\n",
    "\n",
    "myenv.docker.enabled = True\n",
    "\n",
    "myenv.python.conda_dependencies = CondaDependencies.create(conda_packages=['scikit-learn==0.20.3'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Documentation : https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-set-up-training-targets#amlcompute<br>\n",
    "> Pricing : https://azure.microsoft.com/en-us/pricing/details/machine-learning/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "tags": [
     "sample-amlcompute-provision"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating\n",
      "Succeeded....................\n",
      "AmlCompute wait for completion finished\n",
      "\n",
      "Minimum number of nodes requested have been provisioned\n"
     ]
    }
   ],
   "source": [
    "from azureml.core.compute import ComputeTarget, AmlCompute\n",
    "from azureml.core.compute_target import ComputeTargetException\n",
    "\n",
    "# Nom\n",
    "cpu_cluster_name = \"cpucluster\"\n",
    "\n",
    "try:\n",
    "    cpu_cluster = ComputeTarget(workspace=ws, name=cpu_cluster_name)\n",
    "    print('Found existing cluster, use it.')\n",
    "except ComputeTargetException:\n",
    "    compute_config = AmlCompute.provisioning_configuration(vm_size='STANDARD_D2_V2',\n",
    "                                                           min_nodes = 1, #Mettre à 0 pour automatic shutdown\n",
    "                                                           max_nodes = 4)\n",
    "    cpu_cluster = ComputeTarget.create(ws, cpu_cluster_name, compute_config)\n",
    "\n",
    "cpu_cluster.wait_for_completion(show_output=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "AutoML\n",
      "Designer\n",
      "notebooksjupyter\n",
      "cpucluster\n",
      "AutoMLsdk\n"
     ]
    }
   ],
   "source": [
    "#Liste des compute servers disponibles\n",
    "listecomputeservers = ws.compute_targets\n",
    "for liste in listecomputeservers:\n",
    "    print(liste)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'currentNodeCount': 1,\n",
       " 'targetNodeCount': 1,\n",
       " 'nodeStateCounts': {'preparingNodeCount': 0,\n",
       "  'runningNodeCount': 0,\n",
       "  'idleNodeCount': 1,\n",
       "  'unusableNodeCount': 0,\n",
       "  'leavingNodeCount': 0,\n",
       "  'preemptedNodeCount': 0},\n",
       " 'allocationState': 'Steady',\n",
       " 'allocationStateTransitionTime': '2020-09-16T12:49:49.702000+00:00',\n",
       " 'errors': None,\n",
       " 'creationTime': '2020-09-16T12:47:47.086924+00:00',\n",
       " 'modifiedTime': '2020-09-16T12:48:02.858031+00:00',\n",
       " 'provisioningState': 'Succeeded',\n",
       " 'provisioningStateTransitionTime': None,\n",
       " 'scaleSettings': {'minNodeCount': 1,\n",
       "  'maxNodeCount': 4,\n",
       "  'nodeIdleTimeBeforeScaleDown': 'PT120S'},\n",
       " 'vmPriority': 'Dedicated',\n",
       " 'vmSize': 'STANDARD_D2_V2'}"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cpu_cluster.get_status().serialize()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'nodeId': 'tvmps_ceb3add1e43149cd07b65bd3a6ec13e053e808d85aac9d9c37a442fbeef8f515_d',\n",
       "  'port': 50000,\n",
       "  'publicIpAddress': '51.145.188.28',\n",
       "  'privateIpAddress': '10.0.0.4',\n",
       "  'nodeState': 'idle'}]"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Statut du compute server\n",
    "cpu_cluster.list_nodes()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.4 Configuration et exécution du run"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-rwxrwxrwx 1 root root 1538 Apr 15 15:47 train_aml.py\r\n"
     ]
    }
   ],
   "source": [
    "# Fichier Python à exécuter\n",
    "!ls train_aml.py -l"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "# Copyright (c) Microsoft. All rights reserved.\n",
      "# Licensed under the MIT license.\n",
      "\n",
      "from sklearn.datasets import load_diabetes\n",
      "from sklearn.linear_model import Ridge\n",
      "from sklearn.metrics import mean_squared_error\n",
      "from sklearn.model_selection import train_test_split\n",
      "from azureml.core.run import Run\n",
      "from sklearn.externals import joblib\n",
      "import os\n",
      "import numpy as np\n",
      "\n",
      "os.makedirs('./outputs', exist_ok=True)\n",
      "\n",
      "X, y = load_diabetes(return_X_y=True)\n",
      "\n",
      "run = Run.get_context()\n",
      "\n",
      "X_train, X_test, y_train, y_test = train_test_split(X, y,\n",
      "                                                    test_size=0.2,\n",
      "                                                    random_state=0)\n",
      "data = {\"train\": {\"X\": X_train, \"y\": y_train},\n",
      "        \"test\": {\"X\": X_test, \"y\": y_test}}\n",
      "\n",
      "# list of numbers from 0.0 to 1.0 with a 0.05 interval\n",
      "alphas = np.arange(0.0, 1.0, 0.05)\n",
      "\n",
      "for alpha in alphas:\n",
      "    # Use Ridge algorithm to create a regression model\n",
      "    reg = Ridge(alpha=alpha)\n",
      "    reg.fit(data[\"train\"][\"X\"], data[\"train\"][\"y\"])\n",
      "\n",
      "    preds = reg.predict(data[\"test\"][\"X\"])\n",
      "    mse = mean_squared_error(preds, data[\"test\"][\"y\"])\n",
      "    run.log('alpha', alpha)\n",
      "    run.log('mse', mse)\n",
      "\n",
      "    model_file_name = 'ridge_{0:.2f}.pkl'.format(alpha)\n",
      "    # save model in the outputs folder so it automatically get uploaded\n",
      "    with open(model_file_name, \"wb\") as file:\n",
      "        joblib.dump(value=reg, filename=os.path.join('./outputs/',\n",
      "                                                     model_file_name))\n",
      "\n",
      "    print('alpha is {0:.2f}, and mse is {1:0.2f}'.format(alpha, mse))\n",
      "\n"
     ]
    }
   ],
   "source": [
    "#Visu du fichier\n",
    "with open(os.path.join('./train_aml.py'), 'r') as f:\n",
    "    print(f.read())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.core import ScriptRunConfig\n",
    "from azureml.core.runconfig import DEFAULT_CPU_IMAGE\n",
    "\n",
    "src = ScriptRunConfig(source_directory=project_folder, script='train_aml.py')\n",
    "\n",
    "# Set compute target to the one created in previous step\n",
    "src.run_config.target = cpu_cluster.name\n",
    "\n",
    "# Set environment\n",
    "src.run_config.environment = myenv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Définition de tags pour le run\n",
    "tagsdurun = {\"Type\": \"test\" , \"Langage\" : \"Python\" , \"Framework\" : \"Scikit-Learn\", \"Team\" : \"DataScience\" , \"Pays\" : \"France\"}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> C'est parti ! On exécute le run"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table style=\"width:100%\"><tr><th>Experiment</th><th>Id</th><th>Type</th><th>Status</th><th>Details Page</th><th>Docs Page</th></tr><tr><td>Exemple5-amlcompute</td><td>Exemple5-amlcompute_1600260712_5467009b</td><td>azureml.scriptrun</td><td>Queued</td><td><a href=\"https://ml.azure.com/experiments/Exemple5-amlcompute/runs/Exemple5-amlcompute_1600260712_5467009b?wsid=/subscriptions/70b8f39e-8863-49f7-b6ba-34a80799550c/resourcegroups/workshopAML2020-rg/workspaces/workshopAML2020\" target=\"_blank\" rel=\"noopener\">Link to Azure Machine Learning studio</a></td><td><a href=\"https://docs.microsoft.com/en-us/python/api/azureml-core/azureml.core.script_run.ScriptRun?view=azure-ml-py\" target=\"_blank\" rel=\"noopener\">Link to Documentation</a></td></tr></table>"
      ],
      "text/plain": [
       "Run(Experiment: Exemple5-amlcompute,\n",
       "Id: Exemple5-amlcompute_1600260712_5467009b,\n",
       "Type: azureml.scriptrun,\n",
       "Status: Queued)"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Execution run\n",
    "run = experiment.submit(config=src, tags=tagsdurun)\n",
    "run"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.5 Widget disponible pour suivre l'avancement du run"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "cba1611b789549c495603f716ea4f6a4",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "_UserRunWidget(widget_settings={'childWidgetDisplay': 'popup', 'send_telemetry': False, 'log_level': 'INFO', '…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/aml.mini.widget.v1": "{\"status\": \"Completed\", \"workbench_run_details_uri\": \"https://ml.azure.com/experiments/Exemple5-amlcompute/runs/Exemple5-amlcompute_1600260712_5467009b?wsid=/subscriptions/70b8f39e-8863-49f7-b6ba-34a80799550c/resourcegroups/workshopAML2020-rg/workspaces/workshopAML2020\", \"run_id\": \"Exemple5-amlcompute_1600260712_5467009b\", \"run_properties\": {\"run_id\": \"Exemple5-amlcompute_1600260712_5467009b\", \"created_utc\": \"2020-09-16T12:51:54.872434Z\", \"properties\": {\"_azureml.ComputeTargetType\": \"amlcompute\", \"ContentSnapshotId\": \"b4f43b25-93fd-4c23-aa9a-3fda1318cf3d\", \"azureml.git.repository_uri\": \"https://github.com/retkowsky/WorkshopAML2020\", \"mlflow.source.git.repoURL\": \"https://github.com/retkowsky/WorkshopAML2020\", \"azureml.git.branch\": \"master\", \"mlflow.source.git.branch\": \"master\", \"azureml.git.commit\": \"eb05ad565a41b5121d26c6fda4b1c6398a9243d7\", \"mlflow.source.git.commit\": \"eb05ad565a41b5121d26c6fda4b1c6398a9243d7\", \"azureml.git.dirty\": \"True\", \"ProcessInfoFile\": \"azureml-logs/process_info.json\", \"ProcessStatusFile\": \"azureml-logs/process_status.json\"}, \"tags\": {\"Type\": \"test\", \"Langage\": \"Python\", \"Framework\": \"Scikit-Learn\", \"Team\": \"DataScience\", \"Pays\": \"France\", \"_aml_system_ComputeTargetStatus\": \"{\\\"AllocationState\\\":\\\"steady\\\",\\\"PreparingNodeCount\\\":0,\\\"RunningNodeCount\\\":0,\\\"CurrentNodeCount\\\":1}\"}, \"script_name\": null, \"arguments\": null, \"end_time_utc\": \"2020-09-16T13:07:26.378574Z\", \"status\": \"Completed\", \"log_files\": {\"azureml-logs/55_azureml-execution-tvmps_ceb3add1e43149cd07b65bd3a6ec13e053e808d85aac9d9c37a442fbeef8f515_d.txt\": \"https://workshopaml2027584246021.blob.core.windows.net/azureml/ExperimentRun/dcid.Exemple5-amlcompute_1600260712_5467009b/azureml-logs/55_azureml-execution-tvmps_ceb3add1e43149cd07b65bd3a6ec13e053e808d85aac9d9c37a442fbeef8f515_d.txt?sv=2019-02-02&sr=b&sig=fo63aFNCTjGlt8NTAveZ2kGXmLIJ81EgjR9t%2Fv4wOJg%3D&st=2020-09-16T12%3A57%3A12Z&se=2020-09-16T21%3A07%3A12Z&sp=r\", \"azureml-logs/65_job_prep-tvmps_ceb3add1e43149cd07b65bd3a6ec13e053e808d85aac9d9c37a442fbeef8f515_d.txt\": \"https://workshopaml2027584246021.blob.core.windows.net/azureml/ExperimentRun/dcid.Exemple5-amlcompute_1600260712_5467009b/azureml-logs/65_job_prep-tvmps_ceb3add1e43149cd07b65bd3a6ec13e053e808d85aac9d9c37a442fbeef8f515_d.txt?sv=2019-02-02&sr=b&sig=QgdL%2B%2BcJ%2FPvmcRmouiN3o%2BnmjZNoVAKtAegUrCxve6Y%3D&st=2020-09-16T12%3A57%3A12Z&se=2020-09-16T21%3A07%3A12Z&sp=r\", \"azureml-logs/70_driver_log.txt\": \"https://workshopaml2027584246021.blob.core.windows.net/azureml/ExperimentRun/dcid.Exemple5-amlcompute_1600260712_5467009b/azureml-logs/70_driver_log.txt?sv=2019-02-02&sr=b&sig=6iZNGdKh0OANTd8lb2l469PxvVkZV%2BcvKA5zUzBMQ8o%3D&st=2020-09-16T12%3A57%3A12Z&se=2020-09-16T21%3A07%3A12Z&sp=r\", \"azureml-logs/75_job_post-tvmps_ceb3add1e43149cd07b65bd3a6ec13e053e808d85aac9d9c37a442fbeef8f515_d.txt\": \"https://workshopaml2027584246021.blob.core.windows.net/azureml/ExperimentRun/dcid.Exemple5-amlcompute_1600260712_5467009b/azureml-logs/75_job_post-tvmps_ceb3add1e43149cd07b65bd3a6ec13e053e808d85aac9d9c37a442fbeef8f515_d.txt?sv=2019-02-02&sr=b&sig=e7St7dfbXcyHin1HoeFRUxvQMM8k5IjkRYaoJ4nJ1GI%3D&st=2020-09-16T12%3A57%3A12Z&se=2020-09-16T21%3A07%3A12Z&sp=r\", \"azureml-logs/process_info.json\": \"https://workshopaml2027584246021.blob.core.windows.net/azureml/ExperimentRun/dcid.Exemple5-amlcompute_1600260712_5467009b/azureml-logs/process_info.json?sv=2019-02-02&sr=b&sig=PROef3AiCwbVFwXULRNm78KO9L7wnY1a06E9usObaFc%3D&st=2020-09-16T12%3A57%3A12Z&se=2020-09-16T21%3A07%3A12Z&sp=r\", \"azureml-logs/process_status.json\": \"https://workshopaml2027584246021.blob.core.windows.net/azureml/ExperimentRun/dcid.Exemple5-amlcompute_1600260712_5467009b/azureml-logs/process_status.json?sv=2019-02-02&sr=b&sig=VU2WmiQcInvS%2Bw1j3XoKHBCR15kJBg44npuYLT5O9Rc%3D&st=2020-09-16T12%3A57%3A12Z&se=2020-09-16T21%3A07%3A12Z&sp=r\", \"logs/azureml/108_azureml.log\": \"https://workshopaml2027584246021.blob.core.windows.net/azureml/ExperimentRun/dcid.Exemple5-amlcompute_1600260712_5467009b/logs/azureml/108_azureml.log?sv=2019-02-02&sr=b&sig=Bq9lR8fLxZbUNSjpBA%2BARvn9NTZ%2FDBake65DHxjzIMw%3D&st=2020-09-16T12%3A57%3A12Z&se=2020-09-16T21%3A07%3A12Z&sp=r\", \"logs/azureml/job_prep_azureml.log\": \"https://workshopaml2027584246021.blob.core.windows.net/azureml/ExperimentRun/dcid.Exemple5-amlcompute_1600260712_5467009b/logs/azureml/job_prep_azureml.log?sv=2019-02-02&sr=b&sig=pqk3xG8W%2BW3DhxiJJBJsqjOKHyeglANzXL21Je1Y8pk%3D&st=2020-09-16T12%3A57%3A12Z&se=2020-09-16T21%3A07%3A12Z&sp=r\", \"logs/azureml/job_release_azureml.log\": \"https://workshopaml2027584246021.blob.core.windows.net/azureml/ExperimentRun/dcid.Exemple5-amlcompute_1600260712_5467009b/logs/azureml/job_release_azureml.log?sv=2019-02-02&sr=b&sig=ytH9ohu65Y1UW8pZJNJzuYSu%2B86dalfds%2FsGWXMO2hk%3D&st=2020-09-16T12%3A57%3A12Z&se=2020-09-16T21%3A07%3A12Z&sp=r\"}, \"log_groups\": [[\"azureml-logs/process_info.json\", \"azureml-logs/process_status.json\", \"logs/azureml/job_prep_azureml.log\", \"logs/azureml/job_release_azureml.log\"], [\"azureml-logs/55_azureml-execution-tvmps_ceb3add1e43149cd07b65bd3a6ec13e053e808d85aac9d9c37a442fbeef8f515_d.txt\"], [\"azureml-logs/65_job_prep-tvmps_ceb3add1e43149cd07b65bd3a6ec13e053e808d85aac9d9c37a442fbeef8f515_d.txt\"], [\"azureml-logs/70_driver_log.txt\"], [\"azureml-logs/75_job_post-tvmps_ceb3add1e43149cd07b65bd3a6ec13e053e808d85aac9d9c37a442fbeef8f515_d.txt\"], [\"logs/azureml/108_azureml.log\"]], \"run_duration\": \"0:15:31\"}, \"child_runs\": [], \"children_metrics\": {}, \"run_metrics\": [], \"run_logs\": \"2020-09-16 12:55:00,779|azureml|DEBUG|Inputs:: kwargs: {'OutputCollection': True, 'EnableMLflowTracking': True, 'snapshotProject': True, 'only_in_process_features': True, 'skip_track_logs_dir': True}, track_folders: None, deny_list: None, directories_to_watch: []\\n2020-09-16 12:55:00,780|azureml.history._tracking.PythonWorkingDirectory|DEBUG|Execution target type: batchai\\n2020-09-16 12:55:00,789|azureml.history._tracking.PythonWorkingDirectory|DEBUG|Failed to import pyspark with error: No module named 'pyspark'\\n2020-09-16 12:55:00,789|azureml.history._tracking.PythonWorkingDirectory.workingdir|DEBUG|Pinning working directory for filesystems: ['pyfs']\\n2020-09-16 12:55:01,112|azureml.core.run|DEBUG|Adding new factory <function ScriptRun._from_run_dto at 0x7f206ab00400> for run source azureml.scriptrun\\n2020-09-16 12:55:01,143|azureml.core.authentication.TokenRefresherDaemon|DEBUG|Starting daemon and triggering first instance\\n2020-09-16 12:55:01,151|azureml._restclient.clientbase|INFO|Created a worker pool for first use\\n2020-09-16 12:55:01,151|azureml.core.authentication|DEBUG|Time to expire 1814212.848121 seconds\\n2020-09-16 12:55:01,152|azureml._base_sdk_common.service_discovery|DEBUG|Found history service url in environment variable AZUREML_SERVICE_ENDPOINT, history service url: https://westeurope.experiments.azureml.net.\\n2020-09-16 12:55:01,152|azureml._base_sdk_common.service_discovery|DEBUG|Found history service url in environment variable AZUREML_SERVICE_ENDPOINT, history service url: https://westeurope.experiments.azureml.net.\\n2020-09-16 12:55:01,152|azureml._base_sdk_common.service_discovery|DEBUG|Found history service url in environment variable AZUREML_SERVICE_ENDPOINT, history service url: https://westeurope.experiments.azureml.net.\\n2020-09-16 12:55:01,152|azureml._base_sdk_common.service_discovery|DEBUG|Found history service url in environment variable AZUREML_SERVICE_ENDPOINT, history service url: https://westeurope.experiments.azureml.net.\\n2020-09-16 12:55:01,195|azureml._base_sdk_common.service_discovery|DEBUG|Found history service url in environment variable AZUREML_SERVICE_ENDPOINT, history service url: https://westeurope.experiments.azureml.net.\\n2020-09-16 12:55:01,195|azureml._base_sdk_common.service_discovery|DEBUG|Found history service url in environment variable AZUREML_SERVICE_ENDPOINT, history service url: https://westeurope.experiments.azureml.net.\\n2020-09-16 12:55:01,196|azureml._base_sdk_common.service_discovery|DEBUG|Found history service url in environment variable AZUREML_SERVICE_ENDPOINT, history service url: https://westeurope.experiments.azureml.net.\\n2020-09-16 12:55:01,231|azureml._SubmittedRun#Exemple5-amlcompute_1600260712_5467009b.RunHistoryFacade.RunClient.get-async:False|DEBUG|[START]\\n2020-09-16 12:55:01,341|azureml._SubmittedRun#Exemple5-amlcompute_1600260712_5467009b.RunHistoryFacade.RunClient.get-async:False|DEBUG|[STOP]\\n2020-09-16 12:55:01,341|azureml._SubmittedRun#Exemple5-amlcompute_1600260712_5467009b|DEBUG|Constructing run from dto. type: azureml.scriptrun, source: None, props: {'_azureml.ComputeTargetType': 'amlcompute', 'ContentSnapshotId': 'b4f43b25-93fd-4c23-aa9a-3fda1318cf3d', 'azureml.git.repository_uri': 'https://github.com/retkowsky/WorkshopAML2020', 'mlflow.source.git.repoURL': 'https://github.com/retkowsky/WorkshopAML2020', 'azureml.git.branch': 'master', 'mlflow.source.git.branch': 'master', 'azureml.git.commit': 'eb05ad565a41b5121d26c6fda4b1c6398a9243d7', 'mlflow.source.git.commit': 'eb05ad565a41b5121d26c6fda4b1c6398a9243d7', 'azureml.git.dirty': 'True', 'ProcessInfoFile': 'azureml-logs/process_info.json', 'ProcessStatusFile': 'azureml-logs/process_status.json'}\\n2020-09-16 12:55:01,371|azureml._SubmittedRun#Exemple5-amlcompute_1600260712_5467009b.RunContextManager|DEBUG|Valid logs dir, setting up content loader\\n2020-09-16 12:55:01,371|azureml|WARNING|Could not import azureml.mlflow or azureml.contrib.mlflow mlflow APIs will not run against AzureML services.  Add azureml-mlflow as a conda dependency for the run if this behavior is desired\\n2020-09-16 12:55:01,371|azureml.WorkerPool|DEBUG|[START]\\n2020-09-16 12:55:01,372|azureml.SendRunKillSignal|DEBUG|[START]\\n2020-09-16 12:55:01,372|azureml.RunStatusContext|DEBUG|[START]\\n2020-09-16 12:55:01,372|azureml._SubmittedRun#Exemple5-amlcompute_1600260712_5467009b.RunContextManager.RunStatusContext|DEBUG|[START]\\n2020-09-16 12:55:01,372|azureml.WorkingDirectoryCM|DEBUG|[START]\\n2020-09-16 12:55:01,372|azureml.history._tracking.PythonWorkingDirectory.workingdir|DEBUG|[START]\\n2020-09-16 12:55:01,372|azureml.history._tracking.PythonWorkingDirectory|INFO|Current working dir: /mnt/batch/tasks/shared/LS_root/jobs/workshopaml2020/azureml/exemple5-amlcompute_1600260712_5467009b/mounts/workspaceblobstore/azureml/Exemple5-amlcompute_1600260712_5467009b\\n2020-09-16 12:55:01,372|azureml.history._tracking.PythonWorkingDirectory.workingdir|DEBUG|Calling pyfs\\n2020-09-16 12:55:01,372|azureml.history._tracking.PythonWorkingDirectory.workingdir|DEBUG|Storing working dir for pyfs as /mnt/batch/tasks/shared/LS_root/jobs/workshopaml2020/azureml/exemple5-amlcompute_1600260712_5467009b/mounts/workspaceblobstore/azureml/Exemple5-amlcompute_1600260712_5467009b\\n2020-09-16 12:55:02,131|azureml._base_sdk_common.service_discovery|DEBUG|Found history service url in environment variable AZUREML_SERVICE_ENDPOINT, history service url: https://westeurope.experiments.azureml.net.\\n2020-09-16 12:55:02,132|azureml._base_sdk_common.service_discovery|DEBUG|Found history service url in environment variable AZUREML_SERVICE_ENDPOINT, history service url: https://westeurope.experiments.azureml.net.\\n2020-09-16 12:55:02,132|azureml._base_sdk_common.service_discovery|DEBUG|Found history service url in environment variable AZUREML_SERVICE_ENDPOINT, history service url: https://westeurope.experiments.azureml.net.\\n2020-09-16 12:55:02,132|azureml._base_sdk_common.service_discovery|DEBUG|Found history service url in environment variable AZUREML_SERVICE_ENDPOINT, history service url: https://westeurope.experiments.azureml.net.\\n2020-09-16 12:55:02,132|azureml._base_sdk_common.service_discovery|DEBUG|Found history service url in environment variable AZUREML_SERVICE_ENDPOINT, history service url: https://westeurope.experiments.azureml.net.\\n2020-09-16 12:55:02,132|azureml._base_sdk_common.service_discovery|DEBUG|Found history service url in environment variable AZUREML_SERVICE_ENDPOINT, history service url: https://westeurope.experiments.azureml.net.\\n2020-09-16 12:55:02,133|azureml._base_sdk_common.service_discovery|DEBUG|Found history service url in environment variable AZUREML_SERVICE_ENDPOINT, history service url: https://westeurope.experiments.azureml.net.\\n2020-09-16 12:55:02,141|azureml._run_impl.run_history_facade|DEBUG|Created a static thread pool for RunHistoryFacade class\\n2020-09-16 12:55:02,168|azureml._SubmittedRun#Exemple5-amlcompute_1600260712_5467009b.RunHistoryFacade.RunClient.get-async:False|DEBUG|[START]\\n2020-09-16 12:55:02,279|azureml._SubmittedRun#Exemple5-amlcompute_1600260712_5467009b.RunHistoryFacade.RunClient.get-async:False|DEBUG|[STOP]\\n2020-09-16 12:55:02,280|azureml._SubmittedRun#Exemple5-amlcompute_1600260712_5467009b|DEBUG|Constructing run from dto. type: azureml.scriptrun, source: None, props: {'_azureml.ComputeTargetType': 'amlcompute', 'ContentSnapshotId': 'b4f43b25-93fd-4c23-aa9a-3fda1318cf3d', 'azureml.git.repository_uri': 'https://github.com/retkowsky/WorkshopAML2020', 'mlflow.source.git.repoURL': 'https://github.com/retkowsky/WorkshopAML2020', 'azureml.git.branch': 'master', 'mlflow.source.git.branch': 'master', 'azureml.git.commit': 'eb05ad565a41b5121d26c6fda4b1c6398a9243d7', 'mlflow.source.git.commit': 'eb05ad565a41b5121d26c6fda4b1c6398a9243d7', 'azureml.git.dirty': 'True', 'ProcessInfoFile': 'azureml-logs/process_info.json', 'ProcessStatusFile': 'azureml-logs/process_status.json'}\\n2020-09-16 12:55:02,280|azureml._SubmittedRun#Exemple5-amlcompute_1600260712_5467009b.RunContextManager|DEBUG|Valid logs dir, setting up content loader\\n2020-09-16 12:55:02,472|azureml._SubmittedRun#Exemple5-amlcompute_1600260712_5467009b.RunHistoryFacade.MetricsClient|DEBUG|Overrides: Max batch size: 50, batch cushion: 5, Interval: 1.\\n2020-09-16 12:55:02,474|azureml._SubmittedRun#Exemple5-amlcompute_1600260712_5467009b.RunHistoryFacade.MetricsClient.PostMetricsBatchV2.PostMetricsBatchV2Daemon|DEBUG|Starting daemon and triggering first instance\\n2020-09-16 12:55:02,475|azureml._SubmittedRun#Exemple5-amlcompute_1600260712_5467009b.RunHistoryFacade.MetricsClient|DEBUG|Used <class 'azureml._common.async_utils.batch_task_queue.BatchTaskQueue'> for use_batch=True.\\n2020-09-16 12:55:03,475|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|[Start]\\n2020-09-16 12:55:03,475|azureml.BatchTaskQueueAdd_1_Batches.WorkerPool|DEBUG|submitting future: _handle_batch\\n2020-09-16 12:55:03,476|azureml._SubmittedRun#Exemple5-amlcompute_1600260712_5467009b.RunHistoryFacade.MetricsClient.PostMetricsBatchV2|DEBUG|Batch size 18.\\n2020-09-16 12:55:03,476|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch|DEBUG|Using basic handler - no exception handling\\n2020-09-16 12:55:03,478|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|Adding task 0__handle_batch to queue of approximate size: 0\\n2020-09-16 12:55:03,479|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|[Stop] - waiting default timeout\\n2020-09-16 12:55:03,479|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|[START]\\n2020-09-16 12:55:03,479|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|Overriding default flush timeout from None to 120\\n2020-09-16 12:55:03,479|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|Waiting 120 seconds on tasks: [AsyncTask(0__handle_batch)].\\n2020-09-16 12:55:03,478|azureml._restclient.clientbase.WorkerPool|DEBUG|submitting future: _log_batch_v2\\n2020-09-16 12:55:03,480|azureml._SubmittedRun#Exemple5-amlcompute_1600260712_5467009b.RunHistoryFacade.MetricsClient.post_run_metrics-async:False|DEBUG|[START]\\n2020-09-16 12:55:03,488|azureml._SubmittedRun#Exemple5-amlcompute_1600260712_5467009b.RunHistoryFacade.MetricsClient.PostMetricsBatchV2.0__log_batch_v2|DEBUG|Using basic handler - no exception handling\\n2020-09-16 12:55:03,488|azureml._SubmittedRun#Exemple5-amlcompute_1600260712_5467009b.RunHistoryFacade.MetricsClient.PostMetricsBatchV2|DEBUG|Adding task 0__log_batch_v2 to queue of approximate size: 0\\n2020-09-16 12:55:03,729|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch.WaitingTask|DEBUG|[START]\\n2020-09-16 12:55:03,729|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch.WaitingTask|DEBUG|Awaiter is BatchTaskQueueAdd_1_Batches\\n2020-09-16 12:55:03,730|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch.WaitingTask|DEBUG|[STOP]\\n2020-09-16 12:55:03,730|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|Waiting on task: 0__handle_batch.\\n1 tasks left. Current duration of flush 0.00011515617370605469 seconds.\\n\\n2020-09-16 12:55:03,730|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|[STOP]\\n2020-09-16 12:55:03,752|azureml._SubmittedRun#Exemple5-amlcompute_1600260712_5467009b.RunHistoryFacade.MetricsClient.post_run_metrics-async:False|DEBUG|[STOP]\\n2020-09-16 12:55:04,476|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|[Start]\\n2020-09-16 12:55:04,476|azureml.BatchTaskQueueAdd_1_Batches.WorkerPool|DEBUG|submitting future: _handle_batch\\n2020-09-16 12:55:04,476|azureml._SubmittedRun#Exemple5-amlcompute_1600260712_5467009b.RunHistoryFacade.MetricsClient.PostMetricsBatchV2|DEBUG|Batch size 16.\\n2020-09-16 12:55:04,476|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch|DEBUG|Using basic handler - no exception handling\\n2020-09-16 12:55:04,477|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|Adding task 0__handle_batch to queue of approximate size: 0\\n2020-09-16 12:55:04,477|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|[Stop] - waiting default timeout\\n2020-09-16 12:55:04,477|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|[START]\\n2020-09-16 12:55:04,477|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|Overriding default flush timeout from None to 120\\n2020-09-16 12:55:04,477|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|Waiting 120 seconds on tasks: [AsyncTask(0__handle_batch)].\\n2020-09-16 12:55:04,476|azureml._restclient.clientbase.WorkerPool|DEBUG|submitting future: _log_batch_v2\\n2020-09-16 12:55:04,477|azureml._SubmittedRun#Exemple5-amlcompute_1600260712_5467009b.RunHistoryFacade.MetricsClient.post_run_metrics-async:False|DEBUG|[START]\\n2020-09-16 12:55:04,482|azureml._SubmittedRun#Exemple5-amlcompute_1600260712_5467009b.RunHistoryFacade.MetricsClient.PostMetricsBatchV2.1__log_batch_v2|DEBUG|Using basic handler - no exception handling\\n2020-09-16 12:55:04,482|azureml._SubmittedRun#Exemple5-amlcompute_1600260712_5467009b.RunHistoryFacade.MetricsClient.PostMetricsBatchV2|DEBUG|Adding task 1__log_batch_v2 to queue of approximate size: 1\\n2020-09-16 12:55:04,668|azureml._SubmittedRun#Exemple5-amlcompute_1600260712_5467009b.RunHistoryFacade.MetricsClient.post_run_metrics-async:False|DEBUG|[STOP]\\n2020-09-16 12:55:04,728|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch.WaitingTask|DEBUG|[START]\\n2020-09-16 12:55:04,728|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch.WaitingTask|DEBUG|Awaiter is BatchTaskQueueAdd_1_Batches\\n2020-09-16 12:55:04,728|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch.WaitingTask|DEBUG|[STOP]\\n2020-09-16 12:55:04,728|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|Waiting on task: 0__handle_batch.\\n1 tasks left. Current duration of flush 0.00013589859008789062 seconds.\\n\\n2020-09-16 12:55:04,728|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|[STOP]\\n2020-09-16 12:55:05,010|azureml.history._tracking.PythonWorkingDirectory.workingdir|DEBUG|Calling pyfs\\n2020-09-16 12:55:05,010|azureml.history._tracking.PythonWorkingDirectory|INFO|Current working dir: /mnt/batch/tasks/shared/LS_root/jobs/workshopaml2020/azureml/exemple5-amlcompute_1600260712_5467009b/mounts/workspaceblobstore/azureml/Exemple5-amlcompute_1600260712_5467009b\\n2020-09-16 12:55:05,011|azureml.history._tracking.PythonWorkingDirectory.workingdir|DEBUG|Reverting working dir from /mnt/batch/tasks/shared/LS_root/jobs/workshopaml2020/azureml/exemple5-amlcompute_1600260712_5467009b/mounts/workspaceblobstore/azureml/Exemple5-amlcompute_1600260712_5467009b to /mnt/batch/tasks/shared/LS_root/jobs/workshopaml2020/azureml/exemple5-amlcompute_1600260712_5467009b/mounts/workspaceblobstore/azureml/Exemple5-amlcompute_1600260712_5467009b\\n2020-09-16 12:55:05,011|azureml.history._tracking.PythonWorkingDirectory|INFO|Working dir is already updated /mnt/batch/tasks/shared/LS_root/jobs/workshopaml2020/azureml/exemple5-amlcompute_1600260712_5467009b/mounts/workspaceblobstore/azureml/Exemple5-amlcompute_1600260712_5467009b\\n2020-09-16 12:55:05,011|azureml.history._tracking.PythonWorkingDirectory.workingdir|DEBUG|[STOP]\\n2020-09-16 12:55:05,011|azureml.WorkingDirectoryCM|DEBUG|[STOP]\\n2020-09-16 12:55:05,011|azureml._SubmittedRun#Exemple5-amlcompute_1600260712_5467009b|INFO|complete is not setting status for submitted runs.\\n2020-09-16 12:55:05,011|azureml._SubmittedRun#Exemple5-amlcompute_1600260712_5467009b.RunHistoryFacade.MetricsClient.FlushingMetricsClient|DEBUG|[START]\\n2020-09-16 12:55:05,011|azureml._SubmittedRun#Exemple5-amlcompute_1600260712_5467009b.RunHistoryFacade.MetricsClient|DEBUG|Overrides: Max batch size: 50, batch cushion: 5, Interval: 1.\\n2020-09-16 12:55:05,011|azureml._SubmittedRun#Exemple5-amlcompute_1600260712_5467009b.RunHistoryFacade.MetricsClient.PostMetricsBatch.PostMetricsBatchDaemon|DEBUG|Starting daemon and triggering first instance\\n2020-09-16 12:55:05,012|azureml._SubmittedRun#Exemple5-amlcompute_1600260712_5467009b.RunHistoryFacade.MetricsClient|DEBUG|Used <class 'azureml._common.async_utils.batch_task_queue.BatchTaskQueue'> for use_batch=True.\\n2020-09-16 12:55:05,012|azureml._SubmittedRun#Exemple5-amlcompute_1600260712_5467009b.RunHistoryFacade.MetricsClient.PostMetricsBatch.WaitFlushSource:MetricsClient|DEBUG|[START]\\n2020-09-16 12:55:05,012|azureml._SubmittedRun#Exemple5-amlcompute_1600260712_5467009b.RunHistoryFacade.MetricsClient.PostMetricsBatch.WaitFlushSource:MetricsClient|DEBUG|flush timeout 300 is different from task queue timeout 120, using flush timeout\\n2020-09-16 12:55:05,012|azureml._SubmittedRun#Exemple5-amlcompute_1600260712_5467009b.RunHistoryFacade.MetricsClient.PostMetricsBatch.WaitFlushSource:MetricsClient|DEBUG|Waiting 300 seconds on tasks: [].\\n2020-09-16 12:55:05,012|azureml._SubmittedRun#Exemple5-amlcompute_1600260712_5467009b.RunHistoryFacade.MetricsClient.PostMetricsBatch|DEBUG|\\n2020-09-16 12:55:05,012|azureml._SubmittedRun#Exemple5-amlcompute_1600260712_5467009b.RunHistoryFacade.MetricsClient.PostMetricsBatch.WaitFlushSource:MetricsClient|DEBUG|[STOP]\\n2020-09-16 12:55:05,012|azureml._SubmittedRun#Exemple5-amlcompute_1600260712_5467009b.RunHistoryFacade.MetricsClient|DEBUG|Overrides: Max batch size: 50, batch cushion: 5, Interval: 1.\\n2020-09-16 12:55:05,012|azureml._SubmittedRun#Exemple5-amlcompute_1600260712_5467009b.RunHistoryFacade.MetricsClient.PostMetricsBatchV2.PostMetricsBatchV2Daemon|DEBUG|Starting daemon and triggering first instance\\n2020-09-16 12:55:05,013|azureml._SubmittedRun#Exemple5-amlcompute_1600260712_5467009b.RunHistoryFacade.MetricsClient|DEBUG|Used <class 'azureml._common.async_utils.batch_task_queue.BatchTaskQueue'> for use_batch=True.\\n2020-09-16 12:55:05,013|azureml._SubmittedRun#Exemple5-amlcompute_1600260712_5467009b.RunHistoryFacade.MetricsClient.PostMetricsBatchV2.WaitFlushSource:MetricsClient|DEBUG|[START]\\n2020-09-16 12:55:05,013|azureml._SubmittedRun#Exemple5-amlcompute_1600260712_5467009b.RunHistoryFacade.MetricsClient.PostMetricsBatchV2.WaitFlushSource:MetricsClient|DEBUG|flush timeout 300 is different from task queue timeout 120, using flush timeout\\n2020-09-16 12:55:05,013|azureml._SubmittedRun#Exemple5-amlcompute_1600260712_5467009b.RunHistoryFacade.MetricsClient.PostMetricsBatchV2.WaitFlushSource:MetricsClient|DEBUG|Waiting 300 seconds on tasks: [].\\n2020-09-16 12:55:05,013|azureml._SubmittedRun#Exemple5-amlcompute_1600260712_5467009b.RunHistoryFacade.MetricsClient.PostMetricsBatchV2|DEBUG|\\n2020-09-16 12:55:05,013|azureml._SubmittedRun#Exemple5-amlcompute_1600260712_5467009b.RunHistoryFacade.MetricsClient.PostMetricsBatchV2.WaitFlushSource:MetricsClient|DEBUG|[STOP]\\n2020-09-16 12:55:05,013|azureml._SubmittedRun#Exemple5-amlcompute_1600260712_5467009b.RunHistoryFacade.MetricsClient.FlushingMetricsClient|DEBUG|[STOP]\\n2020-09-16 12:55:05,013|azureml._SubmittedRun#Exemple5-amlcompute_1600260712_5467009b.RunHistoryFacade.MetricsClient.wait_on_ingest-async:False|DEBUG|[START]\\n2020-09-16 12:55:05,123|azureml._SubmittedRun#Exemple5-amlcompute_1600260712_5467009b.RunHistoryFacade.MetricsClient.wait_on_ingest-async:False|DEBUG|[STOP]\\n2020-09-16 12:55:05,476|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|[Start]\\n2020-09-16 12:55:05,477|azureml.BatchTaskQueueAdd_1_Batches.WorkerPool|DEBUG|submitting future: _handle_batch\\n2020-09-16 12:55:05,477|azureml._SubmittedRun#Exemple5-amlcompute_1600260712_5467009b.RunHistoryFacade.MetricsClient.PostMetricsBatchV2|DEBUG|Batch size 6.\\n2020-09-16 12:55:05,477|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch|DEBUG|Using basic handler - no exception handling\\n2020-09-16 12:55:05,478|azureml._restclient.clientbase.WorkerPool|DEBUG|submitting future: _log_batch_v2\\n2020-09-16 12:55:05,478|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|Adding task 0__handle_batch to queue of approximate size: 0\\n2020-09-16 12:55:05,478|azureml._SubmittedRun#Exemple5-amlcompute_1600260712_5467009b.RunHistoryFacade.MetricsClient.post_run_metrics-async:False|DEBUG|[START]\\n2020-09-16 12:55:05,478|azureml._SubmittedRun#Exemple5-amlcompute_1600260712_5467009b.RunHistoryFacade.MetricsClient.PostMetricsBatchV2.2__log_batch_v2|DEBUG|Using basic handler - no exception handling\\n2020-09-16 12:55:05,478|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|[Stop] - waiting default timeout\\n2020-09-16 12:55:05,484|azureml._SubmittedRun#Exemple5-amlcompute_1600260712_5467009b.RunHistoryFacade.MetricsClient.PostMetricsBatchV2|DEBUG|Adding task 2__log_batch_v2 to queue of approximate size: 2\\n2020-09-16 12:55:05,485|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|[START]\\n2020-09-16 12:55:05,485|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|Overriding default flush timeout from None to 120\\n2020-09-16 12:55:05,485|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|Waiting 120 seconds on tasks: [AsyncTask(0__handle_batch)].\\n2020-09-16 12:55:05,485|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch.WaitingTask|DEBUG|[START]\\n2020-09-16 12:55:05,485|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch.WaitingTask|DEBUG|Awaiter is BatchTaskQueueAdd_1_Batches\\n2020-09-16 12:55:05,485|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch.WaitingTask|DEBUG|[STOP]\\n2020-09-16 12:55:05,485|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|\\n2020-09-16 12:55:05,486|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|[STOP]\\n2020-09-16 12:55:05,649|azureml._SubmittedRun#Exemple5-amlcompute_1600260712_5467009b.RunHistoryFacade.MetricsClient.post_run_metrics-async:False|DEBUG|[STOP]\\n2020-09-16 12:55:31,144|azureml.core.authentication|DEBUG|Time to expire 1814182.855767 seconds\\n2020-09-16 12:56:01,144|azureml.core.authentication|DEBUG|Time to expire 1814152.855467 seconds\\n2020-09-16 12:56:31,144|azureml.core.authentication|DEBUG|Time to expire 1814122.855154 seconds\\n2020-09-16 12:57:01,145|azureml.core.authentication|DEBUG|Time to expire 1814092.85481 seconds\\n2020-09-16 12:57:31,145|azureml.core.authentication|DEBUG|Time to expire 1814062.854303 seconds\\n2020-09-16 12:58:01,145|azureml.core.authentication|DEBUG|Time to expire 1814032.854055 seconds\\n2020-09-16 12:58:31,146|azureml.core.authentication|DEBUG|Time to expire 1814002.853768 seconds\\n2020-09-16 12:59:01,146|azureml.core.authentication|DEBUG|Time to expire 1813972.853547 seconds\\n2020-09-16 12:59:05,125|azureml.RunStatusContext|DEBUG|[STOP]\\n2020-09-16 12:59:05,125|azureml._SubmittedRun#Exemple5-amlcompute_1600260712_5467009b.RunHistoryFacade.MetricsClient.FlushingMetricsClient|DEBUG|[START]\\n2020-09-16 12:59:05,125|azureml._SubmittedRun#Exemple5-amlcompute_1600260712_5467009b.RunHistoryFacade.MetricsClient.PostMetricsBatch.WaitFlushSource:MetricsClient|DEBUG|[START]\\n2020-09-16 12:59:05,125|azureml._SubmittedRun#Exemple5-amlcompute_1600260712_5467009b.RunHistoryFacade.MetricsClient.PostMetricsBatch.WaitFlushSource:MetricsClient|DEBUG|flush timeout 300.0 is different from task queue timeout 120, using flush timeout\\n2020-09-16 12:59:05,125|azureml._SubmittedRun#Exemple5-amlcompute_1600260712_5467009b.RunHistoryFacade.MetricsClient.PostMetricsBatch.WaitFlushSource:MetricsClient|DEBUG|Waiting 300.0 seconds on tasks: [].\\n2020-09-16 12:59:05,125|azureml._SubmittedRun#Exemple5-amlcompute_1600260712_5467009b.RunHistoryFacade.MetricsClient.PostMetricsBatch|DEBUG|\\n2020-09-16 12:59:05,126|azureml._SubmittedRun#Exemple5-amlcompute_1600260712_5467009b.RunHistoryFacade.MetricsClient.PostMetricsBatch.WaitFlushSource:MetricsClient|DEBUG|[STOP]\\n2020-09-16 12:59:05,126|azureml._SubmittedRun#Exemple5-amlcompute_1600260712_5467009b.RunHistoryFacade.MetricsClient.PostMetricsBatchV2.WaitFlushSource:MetricsClient|DEBUG|[START]\\n2020-09-16 12:59:05,126|azureml._SubmittedRun#Exemple5-amlcompute_1600260712_5467009b.RunHistoryFacade.MetricsClient.PostMetricsBatchV2.WaitFlushSource:MetricsClient|DEBUG|flush timeout 300.0 is different from task queue timeout 120, using flush timeout\\n2020-09-16 12:59:05,126|azureml._SubmittedRun#Exemple5-amlcompute_1600260712_5467009b.RunHistoryFacade.MetricsClient.PostMetricsBatchV2.WaitFlushSource:MetricsClient|DEBUG|Waiting 300.0 seconds on tasks: [].\\n2020-09-16 12:59:05,126|azureml._SubmittedRun#Exemple5-amlcompute_1600260712_5467009b.RunHistoryFacade.MetricsClient.PostMetricsBatchV2|DEBUG|\\n2020-09-16 12:59:05,126|azureml._SubmittedRun#Exemple5-amlcompute_1600260712_5467009b.RunHistoryFacade.MetricsClient.PostMetricsBatchV2.WaitFlushSource:MetricsClient|DEBUG|[STOP]\\n2020-09-16 12:59:05,126|azureml._SubmittedRun#Exemple5-amlcompute_1600260712_5467009b.RunHistoryFacade.MetricsClient.FlushingMetricsClient|DEBUG|[STOP]\\n2020-09-16 12:59:05,126|azureml._SubmittedRun#Exemple5-amlcompute_1600260712_5467009b.RunHistoryFacade.MetricsClient.wait_on_ingest-async:False|DEBUG|[START]\\n2020-09-16 12:59:05,289|azureml._SubmittedRun#Exemple5-amlcompute_1600260712_5467009b.RunHistoryFacade.MetricsClient.wait_on_ingest-async:False|DEBUG|[STOP]\\n2020-09-16 12:59:31,146|azureml.core.authentication|DEBUG|Time to expire 1813942.853269 seconds\\n2020-09-16 13:00:01,147|azureml.core.authentication|DEBUG|Time to expire 1813912.853029 seconds\\n2020-09-16 13:00:31,147|azureml.core.authentication|DEBUG|Time to expire 1813882.852701 seconds\\n2020-09-16 13:01:01,147|azureml.core.authentication|DEBUG|Time to expire 1813852.852368 seconds\\n2020-09-16 13:01:31,147|azureml.core.authentication|DEBUG|Time to expire 1813822.852073 seconds\\n2020-09-16 13:02:01,148|azureml.core.authentication|DEBUG|Time to expire 1813792.851783 seconds\\n2020-09-16 13:02:31,148|azureml.core.authentication|DEBUG|Time to expire 1813762.851426 seconds\\n2020-09-16 13:03:01,148|azureml.core.authentication|DEBUG|Time to expire 1813732.851098 seconds\\n2020-09-16 13:03:05,290|azureml._SubmittedRun#Exemple5-amlcompute_1600260712_5467009b.RunHistoryFacade.MetricsClient.FlushingMetricsClient|DEBUG|[START]\\n2020-09-16 13:03:05,290|azureml._SubmittedRun#Exemple5-amlcompute_1600260712_5467009b.RunHistoryFacade.MetricsClient|DEBUG|Overrides: Max batch size: 50, batch cushion: 5, Interval: 1.\\n2020-09-16 13:03:05,290|azureml._SubmittedRun#Exemple5-amlcompute_1600260712_5467009b.RunHistoryFacade.MetricsClient.PostMetricsBatch.PostMetricsBatchDaemon|DEBUG|Starting daemon and triggering first instance\\n2020-09-16 13:03:05,291|azureml._SubmittedRun#Exemple5-amlcompute_1600260712_5467009b.RunHistoryFacade.MetricsClient|DEBUG|Used <class 'azureml._common.async_utils.batch_task_queue.BatchTaskQueue'> for use_batch=True.\\n2020-09-16 13:03:05,291|azureml._SubmittedRun#Exemple5-amlcompute_1600260712_5467009b.RunHistoryFacade.MetricsClient.PostMetricsBatch.WaitFlushSource:MetricsClient|DEBUG|[START]\\n2020-09-16 13:03:05,291|azureml._SubmittedRun#Exemple5-amlcompute_1600260712_5467009b.RunHistoryFacade.MetricsClient.PostMetricsBatch.WaitFlushSource:MetricsClient|DEBUG|flush timeout 300.0 is different from task queue timeout 120, using flush timeout\\n2020-09-16 13:03:05,291|azureml._SubmittedRun#Exemple5-amlcompute_1600260712_5467009b.RunHistoryFacade.MetricsClient.PostMetricsBatch.WaitFlushSource:MetricsClient|DEBUG|Waiting 300.0 seconds on tasks: [].\\n2020-09-16 13:03:05,291|azureml._SubmittedRun#Exemple5-amlcompute_1600260712_5467009b.RunHistoryFacade.MetricsClient.PostMetricsBatch|DEBUG|\\n2020-09-16 13:03:05,291|azureml._SubmittedRun#Exemple5-amlcompute_1600260712_5467009b.RunHistoryFacade.MetricsClient.PostMetricsBatch.WaitFlushSource:MetricsClient|DEBUG|[STOP]\\n2020-09-16 13:03:05,291|azureml._SubmittedRun#Exemple5-amlcompute_1600260712_5467009b.RunHistoryFacade.MetricsClient.PostMetricsBatchV2.WaitFlushSource:MetricsClient|DEBUG|[START]\\n2020-09-16 13:03:05,291|azureml._SubmittedRun#Exemple5-amlcompute_1600260712_5467009b.RunHistoryFacade.MetricsClient.PostMetricsBatchV2.WaitFlushSource:MetricsClient|DEBUG|flush timeout 300.0 is different from task queue timeout 120, using flush timeout\\n2020-09-16 13:03:05,292|azureml._SubmittedRun#Exemple5-amlcompute_1600260712_5467009b.RunHistoryFacade.MetricsClient.PostMetricsBatchV2.WaitFlushSource:MetricsClient|DEBUG|Waiting 300.0 seconds on tasks: [AsyncTask(0__log_batch_v2), AsyncTask(1__log_batch_v2), AsyncTask(2__log_batch_v2)].\\n2020-09-16 13:03:05,292|azureml._SubmittedRun#Exemple5-amlcompute_1600260712_5467009b.RunHistoryFacade.MetricsClient.PostMetricsBatchV2.0__log_batch_v2.WaitingTask|DEBUG|[START]\\n2020-09-16 13:03:05,292|azureml._SubmittedRun#Exemple5-amlcompute_1600260712_5467009b.RunHistoryFacade.MetricsClient.PostMetricsBatchV2.0__log_batch_v2.WaitingTask|DEBUG|Awaiter is PostMetricsBatchV2\\n2020-09-16 13:03:05,292|azureml._SubmittedRun#Exemple5-amlcompute_1600260712_5467009b.RunHistoryFacade.MetricsClient.PostMetricsBatchV2.0__log_batch_v2.WaitingTask|DEBUG|[STOP]\\n2020-09-16 13:03:05,292|azureml._SubmittedRun#Exemple5-amlcompute_1600260712_5467009b.RunHistoryFacade.MetricsClient.PostMetricsBatchV2.1__log_batch_v2.WaitingTask|DEBUG|[START]\\n2020-09-16 13:03:05,292|azureml._SubmittedRun#Exemple5-amlcompute_1600260712_5467009b.RunHistoryFacade.MetricsClient.PostMetricsBatchV2.1__log_batch_v2.WaitingTask|DEBUG|Awaiter is PostMetricsBatchV2\\n2020-09-16 13:03:05,292|azureml._SubmittedRun#Exemple5-amlcompute_1600260712_5467009b.RunHistoryFacade.MetricsClient.PostMetricsBatchV2.1__log_batch_v2.WaitingTask|DEBUG|[STOP]\\n2020-09-16 13:03:05,292|azureml._SubmittedRun#Exemple5-amlcompute_1600260712_5467009b.RunHistoryFacade.MetricsClient.PostMetricsBatchV2.2__log_batch_v2.WaitingTask|DEBUG|[START]\\n2020-09-16 13:03:05,292|azureml._SubmittedRun#Exemple5-amlcompute_1600260712_5467009b.RunHistoryFacade.MetricsClient.PostMetricsBatchV2.2__log_batch_v2.WaitingTask|DEBUG|Awaiter is PostMetricsBatchV2\\n2020-09-16 13:03:05,292|azureml._SubmittedRun#Exemple5-amlcompute_1600260712_5467009b.RunHistoryFacade.MetricsClient.PostMetricsBatchV2.2__log_batch_v2.WaitingTask|DEBUG|[STOP]\\n2020-09-16 13:03:05,293|azureml._SubmittedRun#Exemple5-amlcompute_1600260712_5467009b.RunHistoryFacade.MetricsClient.PostMetricsBatchV2|DEBUG|\\n2020-09-16 13:03:05,293|azureml._SubmittedRun#Exemple5-amlcompute_1600260712_5467009b.RunHistoryFacade.MetricsClient.PostMetricsBatchV2.WaitFlushSource:MetricsClient|DEBUG|[STOP]\\n2020-09-16 13:03:05,293|azureml._SubmittedRun#Exemple5-amlcompute_1600260712_5467009b.RunHistoryFacade.MetricsClient.FlushingMetricsClient|DEBUG|[STOP]\\n2020-09-16 13:03:05,293|azureml._SubmittedRun#Exemple5-amlcompute_1600260712_5467009b.RunHistoryFacade.MetricsClient.wait_on_ingest-async:False|DEBUG|[START]\\n2020-09-16 13:03:05,407|azureml._SubmittedRun#Exemple5-amlcompute_1600260712_5467009b.RunHistoryFacade.MetricsClient.wait_on_ingest-async:False|DEBUG|[STOP]\\n2020-09-16 13:03:31,149|azureml.core.authentication|DEBUG|Time to expire 1813702.850782 seconds\\n2020-09-16 13:04:01,149|azureml.core.authentication|DEBUG|Time to expire 1813672.850433 seconds\\n2020-09-16 13:04:31,149|azureml.core.authentication|DEBUG|Time to expire 1813642.850066 seconds\\n2020-09-16 13:05:01,150|azureml.core.authentication|DEBUG|Time to expire 1813612.849667 seconds\\n2020-09-16 13:05:31,150|azureml.core.authentication|DEBUG|Time to expire 1813582.849338 seconds\\n2020-09-16 13:06:01,150|azureml.core.authentication|DEBUG|Time to expire 1813552.849074 seconds\\n2020-09-16 13:06:31,151|azureml.core.authentication|DEBUG|Time to expire 1813522.848721 seconds\\n2020-09-16 13:07:01,151|azureml.core.authentication|DEBUG|Time to expire 1813492.848459 seconds\\n2020-09-16 13:07:05,409|azureml.SendRunKillSignal|DEBUG|[STOP]\\n2020-09-16 13:07:05,409|azureml.HistoryTrackingWorkerPool.WorkerPoolShutdown|DEBUG|[START]\\n2020-09-16 13:07:05,409|azureml.HistoryTrackingWorkerPool.WorkerPoolShutdown|DEBUG|[STOP]\\n2020-09-16 13:07:05,409|azureml.WorkerPool|DEBUG|[STOP]\\n\\nRun is completed.\", \"graph\": {}, \"widget_settings\": {\"childWidgetDisplay\": \"popup\", \"send_telemetry\": false, \"log_level\": \"INFO\", \"sdk_version\": \"1.13.0\"}, \"loading\": false}"
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from azureml.widgets import RunDetails\n",
    "RunDetails(run).show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.6 Informations additionnelles"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> **run.get_details** pour suivre **l'avancement du run**. <br>Si le cluster est inactif, cela peut nécessiter + de temps de traitement."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Completed'"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Statut du run\n",
    "run.get_status()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'runId': 'Exemple5-amlcompute_1600260712_5467009b',\n",
       " 'target': 'cpucluster',\n",
       " 'status': 'Completed',\n",
       " 'startTimeUtc': '2020-09-16T12:52:06.368704Z',\n",
       " 'endTimeUtc': '2020-09-16T13:07:26.378574Z',\n",
       " 'properties': {'_azureml.ComputeTargetType': 'amlcompute',\n",
       "  'ContentSnapshotId': 'b4f43b25-93fd-4c23-aa9a-3fda1318cf3d',\n",
       "  'azureml.git.repository_uri': 'https://github.com/retkowsky/WorkshopAML2020',\n",
       "  'mlflow.source.git.repoURL': 'https://github.com/retkowsky/WorkshopAML2020',\n",
       "  'azureml.git.branch': 'master',\n",
       "  'mlflow.source.git.branch': 'master',\n",
       "  'azureml.git.commit': 'eb05ad565a41b5121d26c6fda4b1c6398a9243d7',\n",
       "  'mlflow.source.git.commit': 'eb05ad565a41b5121d26c6fda4b1c6398a9243d7',\n",
       "  'azureml.git.dirty': 'True',\n",
       "  'ProcessInfoFile': 'azureml-logs/process_info.json',\n",
       "  'ProcessStatusFile': 'azureml-logs/process_status.json'},\n",
       " 'inputDatasets': [],\n",
       " 'outputDatasets': [],\n",
       " 'runDefinition': {'script': 'train_aml.py',\n",
       "  'scriptType': None,\n",
       "  'useAbsolutePath': False,\n",
       "  'arguments': [],\n",
       "  'sourceDirectoryDataStore': None,\n",
       "  'framework': 'Python',\n",
       "  'communicator': 'None',\n",
       "  'target': 'cpucluster',\n",
       "  'dataReferences': {},\n",
       "  'data': {},\n",
       "  'outputData': {},\n",
       "  'jobName': None,\n",
       "  'maxRunDurationSeconds': 2592000,\n",
       "  'nodeCount': 1,\n",
       "  'environment': {'name': 'myenv',\n",
       "   'version': 'Autosave_2020-09-09T13:08:45Z_471df787',\n",
       "   'python': {'interpreterPath': 'python',\n",
       "    'userManagedDependencies': False,\n",
       "    'condaDependencies': {'channels': ['anaconda', 'conda-forge'],\n",
       "     'dependencies': ['python=3.6.2',\n",
       "      {'pip': ['azureml-defaults~=1.13.0']},\n",
       "      'scikit-learn==0.20.3'],\n",
       "     'name': 'azureml_765c400a9dc90a956d5e2cd3b6a22dda'},\n",
       "    'baseCondaEnvironment': None},\n",
       "   'environmentVariables': {'EXAMPLE_ENV_VAR': 'EXAMPLE_VALUE'},\n",
       "   'docker': {'baseImage': 'mcr.microsoft.com/azureml/intelmpi2018.3-ubuntu16.04:20200821.v1',\n",
       "    'platform': {'os': 'Linux', 'architecture': 'amd64'},\n",
       "    'baseDockerfile': None,\n",
       "    'baseImageRegistry': {'address': None, 'username': None, 'password': None},\n",
       "    'enabled': True,\n",
       "    'arguments': []},\n",
       "   'spark': {'repositories': [], 'packages': [], 'precachePackages': True},\n",
       "   'inferencingStackVersion': None},\n",
       "  'history': {'outputCollection': True,\n",
       "   'directoriesToWatch': ['logs'],\n",
       "   'enableMLflowTracking': True,\n",
       "   'snapshotProject': True},\n",
       "  'spark': {'configuration': {'spark.app.name': 'Azure ML Experiment',\n",
       "    'spark.yarn.maxAppAttempts': '1'}},\n",
       "  'parallelTask': {'maxRetriesPerWorker': 0,\n",
       "   'workerCountPerNode': 1,\n",
       "   'terminalExitCodes': None,\n",
       "   'configuration': {}},\n",
       "  'amlCompute': {'name': None,\n",
       "   'vmSize': None,\n",
       "   'retainCluster': False,\n",
       "   'clusterMaxNodeCount': None},\n",
       "  'aiSuperComputer': {'instanceType': None,\n",
       "   'frameworkImage': None,\n",
       "   'imageVersion': None,\n",
       "   'location': None},\n",
       "  'tensorflow': {'workerCount': 1, 'parameterServerCount': 1},\n",
       "  'mpi': {'processCountPerNode': 1},\n",
       "  'hdi': {'yarnDeployMode': 'Cluster'},\n",
       "  'containerInstance': {'region': None, 'cpuCores': 2, 'memoryGb': 3.5},\n",
       "  'exposedPorts': None,\n",
       "  'docker': {'useDocker': True,\n",
       "   'sharedVolumes': True,\n",
       "   'shmSize': '2g',\n",
       "   'arguments': []},\n",
       "  'cmk8sCompute': {'configuration': {}},\n",
       "  'cmAksCompute': {'configuration': {}}},\n",
       " 'logFiles': {'azureml-logs/55_azureml-execution-tvmps_ceb3add1e43149cd07b65bd3a6ec13e053e808d85aac9d9c37a442fbeef8f515_d.txt': 'https://workshopaml2027584246021.blob.core.windows.net/azureml/ExperimentRun/dcid.Exemple5-amlcompute_1600260712_5467009b/azureml-logs/55_azureml-execution-tvmps_ceb3add1e43149cd07b65bd3a6ec13e053e808d85aac9d9c37a442fbeef8f515_d.txt?sv=2019-02-02&sr=b&sig=fo63aFNCTjGlt8NTAveZ2kGXmLIJ81EgjR9t%2Fv4wOJg%3D&st=2020-09-16T12%3A57%3A12Z&se=2020-09-16T21%3A07%3A12Z&sp=r',\n",
       "  'azureml-logs/65_job_prep-tvmps_ceb3add1e43149cd07b65bd3a6ec13e053e808d85aac9d9c37a442fbeef8f515_d.txt': 'https://workshopaml2027584246021.blob.core.windows.net/azureml/ExperimentRun/dcid.Exemple5-amlcompute_1600260712_5467009b/azureml-logs/65_job_prep-tvmps_ceb3add1e43149cd07b65bd3a6ec13e053e808d85aac9d9c37a442fbeef8f515_d.txt?sv=2019-02-02&sr=b&sig=QgdL%2B%2BcJ%2FPvmcRmouiN3o%2BnmjZNoVAKtAegUrCxve6Y%3D&st=2020-09-16T12%3A57%3A12Z&se=2020-09-16T21%3A07%3A12Z&sp=r',\n",
       "  'azureml-logs/70_driver_log.txt': 'https://workshopaml2027584246021.blob.core.windows.net/azureml/ExperimentRun/dcid.Exemple5-amlcompute_1600260712_5467009b/azureml-logs/70_driver_log.txt?sv=2019-02-02&sr=b&sig=6iZNGdKh0OANTd8lb2l469PxvVkZV%2BcvKA5zUzBMQ8o%3D&st=2020-09-16T12%3A57%3A12Z&se=2020-09-16T21%3A07%3A12Z&sp=r',\n",
       "  'azureml-logs/75_job_post-tvmps_ceb3add1e43149cd07b65bd3a6ec13e053e808d85aac9d9c37a442fbeef8f515_d.txt': 'https://workshopaml2027584246021.blob.core.windows.net/azureml/ExperimentRun/dcid.Exemple5-amlcompute_1600260712_5467009b/azureml-logs/75_job_post-tvmps_ceb3add1e43149cd07b65bd3a6ec13e053e808d85aac9d9c37a442fbeef8f515_d.txt?sv=2019-02-02&sr=b&sig=e7St7dfbXcyHin1HoeFRUxvQMM8k5IjkRYaoJ4nJ1GI%3D&st=2020-09-16T12%3A57%3A12Z&se=2020-09-16T21%3A07%3A12Z&sp=r',\n",
       "  'azureml-logs/process_info.json': 'https://workshopaml2027584246021.blob.core.windows.net/azureml/ExperimentRun/dcid.Exemple5-amlcompute_1600260712_5467009b/azureml-logs/process_info.json?sv=2019-02-02&sr=b&sig=PROef3AiCwbVFwXULRNm78KO9L7wnY1a06E9usObaFc%3D&st=2020-09-16T12%3A57%3A12Z&se=2020-09-16T21%3A07%3A12Z&sp=r',\n",
       "  'azureml-logs/process_status.json': 'https://workshopaml2027584246021.blob.core.windows.net/azureml/ExperimentRun/dcid.Exemple5-amlcompute_1600260712_5467009b/azureml-logs/process_status.json?sv=2019-02-02&sr=b&sig=VU2WmiQcInvS%2Bw1j3XoKHBCR15kJBg44npuYLT5O9Rc%3D&st=2020-09-16T12%3A57%3A12Z&se=2020-09-16T21%3A07%3A12Z&sp=r',\n",
       "  'logs/azureml/108_azureml.log': 'https://workshopaml2027584246021.blob.core.windows.net/azureml/ExperimentRun/dcid.Exemple5-amlcompute_1600260712_5467009b/logs/azureml/108_azureml.log?sv=2019-02-02&sr=b&sig=Bq9lR8fLxZbUNSjpBA%2BARvn9NTZ%2FDBake65DHxjzIMw%3D&st=2020-09-16T12%3A57%3A12Z&se=2020-09-16T21%3A07%3A12Z&sp=r',\n",
       "  'logs/azureml/job_prep_azureml.log': 'https://workshopaml2027584246021.blob.core.windows.net/azureml/ExperimentRun/dcid.Exemple5-amlcompute_1600260712_5467009b/logs/azureml/job_prep_azureml.log?sv=2019-02-02&sr=b&sig=pqk3xG8W%2BW3DhxiJJBJsqjOKHyeglANzXL21Je1Y8pk%3D&st=2020-09-16T12%3A57%3A12Z&se=2020-09-16T21%3A07%3A12Z&sp=r',\n",
       "  'logs/azureml/job_release_azureml.log': 'https://workshopaml2027584246021.blob.core.windows.net/azureml/ExperimentRun/dcid.Exemple5-amlcompute_1600260712_5467009b/logs/azureml/job_release_azureml.log?sv=2019-02-02&sr=b&sig=ytH9ohu65Y1UW8pZJNJzuYSu%2B86dalfds%2FsGWXMO2hk%3D&st=2020-09-16T12%3A57%3A12Z&se=2020-09-16T21%3A07%3A12Z&sp=r'}}"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Détails du run\n",
    "run.get_details()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'nodeId': 'tvmps_ceb3add1e43149cd07b65bd3a6ec13e053e808d85aac9d9c37a442fbeef8f515_d',\n",
       "  'port': 50000,\n",
       "  'publicIpAddress': '51.145.188.28',\n",
       "  'privateIpAddress': '10.0.0.4',\n",
       "  'nodeState': 'idle'}]"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Statut\n",
    "cpu_cluster.list_nodes()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Pour voir les métriques de l'expérimentation (uniquement en fin de run). <br>Les métriques sont aussi visibles dans le portail Azure."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Liste des métriques :\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{}"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(\"Liste des métriques :\")\n",
    "run.get_metrics()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Informations sur le compute server:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Status du cluster :\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'currentNodeCount': 1,\n",
       " 'targetNodeCount': 1,\n",
       " 'nodeStateCounts': {'preparingNodeCount': 0,\n",
       "  'runningNodeCount': 0,\n",
       "  'idleNodeCount': 1,\n",
       "  'unusableNodeCount': 0,\n",
       "  'leavingNodeCount': 0,\n",
       "  'preemptedNodeCount': 0},\n",
       " 'allocationState': 'Steady',\n",
       " 'allocationStateTransitionTime': '2020-09-16T12:49:49.702000+00:00',\n",
       " 'errors': None,\n",
       " 'creationTime': '2020-09-16T12:47:47.086924+00:00',\n",
       " 'modifiedTime': '2020-09-16T12:48:02.858031+00:00',\n",
       " 'provisioningState': 'Succeeded',\n",
       " 'provisioningStateTransitionTime': None,\n",
       " 'scaleSettings': {'minNodeCount': 1,\n",
       "  'maxNodeCount': 4,\n",
       "  'nodeIdleTimeBeforeScaleDown': 'PT120S'},\n",
       " 'vmPriority': 'Dedicated',\n",
       " 'vmSize': 'STANDARD_D2_V2'}"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(\"Status du cluster :\")\n",
    "cpu_cluster.get_status().serialize()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Noeuds du cluster :\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[{'nodeId': 'tvmps_ceb3add1e43149cd07b65bd3a6ec13e053e808d85aac9d9c37a442fbeef8f515_d',\n",
       "  'port': 50000,\n",
       "  'publicIpAddress': '51.145.188.28',\n",
       "  'privateIpAddress': '10.0.0.4',\n",
       "  'nodeState': 'idle'}]"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(\"Noeuds du cluster :\")\n",
    "cpu_cluster.list_nodes()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### On peut changer la configuration du compute server :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "#cpu_cluster.update(min_nodes=0) # On passe à 0 min node"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Status du cluster\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'currentNodeCount': 1,\n",
       " 'targetNodeCount': 1,\n",
       " 'nodeStateCounts': {'preparingNodeCount': 0,\n",
       "  'runningNodeCount': 0,\n",
       "  'idleNodeCount': 1,\n",
       "  'unusableNodeCount': 0,\n",
       "  'leavingNodeCount': 0,\n",
       "  'preemptedNodeCount': 0},\n",
       " 'allocationState': 'Steady',\n",
       " 'allocationStateTransitionTime': '2020-09-16T12:49:49.702000+00:00',\n",
       " 'errors': None,\n",
       " 'creationTime': '2020-09-16T12:47:47.086924+00:00',\n",
       " 'modifiedTime': '2020-09-16T12:48:02.858031+00:00',\n",
       " 'provisioningState': 'Succeeded',\n",
       " 'provisioningStateTransitionTime': None,\n",
       " 'scaleSettings': {'minNodeCount': 1,\n",
       "  'maxNodeCount': 4,\n",
       "  'nodeIdleTimeBeforeScaleDown': 'PT120S'},\n",
       " 'vmPriority': 'Dedicated',\n",
       " 'vmSize': 'STANDARD_D2_V2'}"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(\"Status du cluster\")\n",
    "cpu_cluster.get_status().serialize()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "#cpu_cluster.update(max_nodes=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "#cpu_cluster.update(idle_seconds_before_scaledown=1200) # On change le timeout"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Status du cluster\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'currentNodeCount': 1,\n",
       " 'targetNodeCount': 1,\n",
       " 'nodeStateCounts': {'preparingNodeCount': 0,\n",
       "  'runningNodeCount': 0,\n",
       "  'idleNodeCount': 1,\n",
       "  'unusableNodeCount': 0,\n",
       "  'leavingNodeCount': 0,\n",
       "  'preemptedNodeCount': 0},\n",
       " 'allocationState': 'Steady',\n",
       " 'allocationStateTransitionTime': '2020-09-16T12:49:49.702000+00:00',\n",
       " 'errors': None,\n",
       " 'creationTime': '2020-09-16T12:47:47.086924+00:00',\n",
       " 'modifiedTime': '2020-09-16T12:48:02.858031+00:00',\n",
       " 'provisioningState': 'Succeeded',\n",
       " 'provisioningStateTransitionTime': None,\n",
       " 'scaleSettings': {'minNodeCount': 1,\n",
       "  'maxNodeCount': 4,\n",
       "  'nodeIdleTimeBeforeScaleDown': 'PT120S'},\n",
       " 'vmPriority': 'Dedicated',\n",
       " 'vmSize': 'STANDARD_D2_V2'}"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(\"Status du cluster\")\n",
    "cpu_cluster.get_status().serialize()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "#cpu_cluster.update(min_nodes=2, max_nodes=4, idle_seconds_before_scaledown=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Suppression du compute server :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Pour supprimer le compute server\n",
    "#cpu_cluster.delete()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "AutoML  -  AmlCompute  -  Succeeded\n",
      "Designer  -  AmlCompute  -  Succeeded\n",
      "notebooksjupyter  -  ComputeInstance  -  Succeeded\n",
      "cpucluster  -  AmlCompute  -  Succeeded\n",
      "AutoMLsdk  -  AmlCompute  -  Succeeded\n",
      "cpupipelines  -  AmlCompute  -  Succeeded\n",
      "clustergpuNC6  -  AmlCompute  -  Succeeded\n"
     ]
    }
   ],
   "source": [
    "compute_targets = ws.compute_targets\n",
    "for name, ct in compute_targets.items():\n",
    "    print(name, \" - \" , ct.type, \" - \", ct.provisioning_state)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"https://github.com/retkowsky/images/blob/master/Powered-by-MS-Azure-logo-v2.png?raw=true\" height=\"300\" width=\"300\">"
   ]
  }
 ],
 "metadata": {
  "authors": [
   {
    "name": "nigup"
   }
  ],
  "category": "training",
  "compute": [
   "AML Compute"
  ],
  "datasets": [
   "Diabetes"
  ],
  "deployment": [
   "None"
  ],
  "exclude_from_index": false,
  "framework": [
   "None"
  ],
  "friendly_name": "Train on Azure Machine Learning Compute",
  "index_order": 1,
  "kernelspec": {
   "display_name": "Python 3.6 - AzureML",
   "language": "python",
   "name": "python3-azureml"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  },
  "tags": [
   "None"
  ],
  "task": "Submit a run on Azure Machine Learning Compute."
 },
 "nbformat": 4,
 "nbformat_minor": 2
}