Skip to content

Commit

Permalink
Samples for AutoML - sweep over pipeline components (#2438)
Browse files Browse the repository at this point in the history 
* Add multi-class notebook

* Add rest of image notebooks

* Minor fixes

* Address comments

* Use curated models from OD

* Use curated models for IS

* Update classification notebooks to use curated models

* Remove mask rcnn model from OD notebook

* Fix typo

* Update code to fetch registry models. Update space

* Update multilabel search space. Add request settings to all notebooks

* Increase sweep timeout to 4 hours

* Change text from model algorithm to model

* Update search space

* Add try/except for model fetching from registry

* Fix formatting

* Fix formatting issues

* Add log message for exception
  • Loading branch information
@PhaniShekhar
PhaniShekhar authored Aug 22, 2023
1 parent 6b4123d commit 4df5a43
Show file tree
Hide file tree
Showing 4 changed files with 715 additions and 46 deletions.
175 changes: 172 additions & 3 deletions ...lticlass-task-fridge-items/automl-image-classification-multiclass-task-fridge-items.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -252,7 +252,6 @@
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
Expand All @@ -279,7 +278,6 @@
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
Expand Down Expand Up @@ -638,6 +636,89 @@
"ml_client.jobs.stream(returned_job.name)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 4.2.1 Individual runs with models from Hugging Face (Preview)\n",
"\n",
"In addition to the models supported natively by AutoML, you can launch individual runs to explore any model from HuggingFace transformers library that supports image classification. Please refer to this [documentation](https://huggingface.co/models?pipeline_tag=image-classification&library=transformers) for the list of models.\n",
"\n",
"While you can use any model from Hugging face to support this task, we have curated a set of models in our registry. We provide a set of sensible default hyperparameters for these models. You can fetch the list of curated models using code snippet below."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"registry_ml_client = MLClient(credential, registry_name=\"azureml\")\n",
"\n",
"models = registry_ml_client.models.list()\n",
"classification_models = []\n",
"for model in models:\n",
" try:\n",
" model = registry_ml_client.models.get(model.name, label=\"latest\")\n",
" if model.tags.get(\"task\", \"\") == \"image-classification\":\n",
" classification_models.append(model.name)\n",
" except Exception as ex:\n",
" print(f\"Error while accessing registry model list: {ex}\")\n",
"\n",
"classification_models"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"If you wish to try a model (say `microsoft/beit-base-patch16-224-pt22k-ft22k`), you can specify the job for your AutoML Image runs as follows:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Create the AutoML job with the related factory-function.\n",
"\n",
"image_classification_job = automl.image_classification(\n",
" compute=compute_name,\n",
" experiment_name=exp_name,\n",
" training_data=my_training_data_input,\n",
" validation_data=my_validation_data_input,\n",
" target_column_name=\"label\",\n",
")\n",
"\n",
"image_classification_job.set_limits(timeout_minutes=60)\n",
"\n",
"image_classification_job.set_training_parameters(\n",
" model_name=\"microsoft/beit-base-patch16-224-pt22k-ft22k\"\n",
")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Submit the AutoML job\n",
"returned_job = ml_client.jobs.create_or_update(image_classification_job)\n",
"\n",
"print(f\"Created job: {returned_job}\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"ml_client.jobs.stream(returned_job.name)"
]
},
{
"cell_type": "markdown",
"metadata": {},
Expand Down Expand Up @@ -774,6 +855,91 @@
"hd_job"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 4.3.1 Manual hyperparameter sweeping for models from Hugging Face (Preview)\n",
"\n",
"Similar to how you can use any model from Hugging face transformers library for individual runs, you can also include these models to perform a hyperparameter sweep. You can also choose a combination of models supported supported natively by [AutoML](https://docs.microsoft.com/en-us/azure/machine-learning/how-to-auto-train-image-models?tabs=CLI-v2#configure-model-algorithms-and-hyperparameters) and models from [Hugging Face](https://huggingface.co/models?pipeline_tag=image-classification&library=transformers).\n",
"\n",
"In this example, we sweep over `microsoft/beit-base-patch16-224-pt22k-ft22k`, `facebook/deit-base-patch16-224`, `seresnext`, and `resnet50` models choosing from a range of values for learning_rate, number_of_epochs, etc., to generate a model with the optimal 'accuracy'."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Create the AutoML job with the related factory-function.\n",
"\n",
"image_classification_job = automl.image_classification(\n",
" compute=compute_name,\n",
" experiment_name=exp_name,\n",
" training_data=my_training_data_input,\n",
" validation_data=my_validation_data_input,\n",
" target_column_name=\"label\",\n",
" primary_metric=ClassificationPrimaryMetrics.ACCURACY,\n",
" tags={\"my_custom_tag\": \"My custom value\"},\n",
")\n",
"\n",
"image_classification_job.set_limits(\n",
" timeout_minutes=240,\n",
" max_trials=10,\n",
" max_concurrent_trials=2,\n",
")\n",
"\n",
"image_classification_job.extend_search_space(\n",
" [\n",
" SearchSpace(\n",
" model_name=Choice(\n",
" [\n",
" \"microsoft/beit-base-patch16-224-pt22k-ft22k\",\n",
" \"facebook/deit-base-patch16-224\",\n",
" ]\n",
" ),\n",
" learning_rate=Uniform(0.00001, 0.0001),\n",
" number_of_epochs=Choice([10, 15]),\n",
" ),\n",
" SearchSpace(\n",
" model_name=Choice([\"seresnext\", \"resnet50\"]),\n",
" learning_rate=Uniform(0.001, 0.01),\n",
" ),\n",
" ]\n",
")\n",
"\n",
"image_classification_job.set_sweep(\n",
" sampling_algorithm=\"Random\",\n",
" early_termination=BanditPolicy(\n",
" evaluation_interval=2, slack_factor=0.2, delay_evaluation=6\n",
" ),\n",
")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Submit the AutoML job\n",
"returned_job_hf = ml_client.jobs.create_or_update(\n",
" image_classification_job\n",
") # submit the job to the backend\n",
"\n",
"print(f\"Created job: {returned_job_hf}\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"ml_client.jobs.stream(returned_job_hf.name)"
]
},
{
"cell_type": "markdown",
"metadata": {},
Expand Down Expand Up @@ -1093,6 +1259,7 @@
"source": [
"from azure.ai.ml.entities import OnlineRequestSettings\n",
"\n",
"# Setting the request timeout to 90 seconds.\n",
"req_timeout = OnlineRequestSettings(request_timeout_ms=90000)"
]
},
Expand Down Expand Up @@ -1285,7 +1452,9 @@
" - [Guided GradCAM](https://arxiv.org/abs/1610.02391v4) (guided_gradcam)\n",
" - [Guided BackPropagation](https://arxiv.org/abs/1412.6806) (guided_backprop)\n",
"\n",
"For more details on explainability with AutoML for images, refer to the [generating explanations](https://learn.microsoft.com/en-us/azure/machine-learning/how-to-auto-train-image-models?tabs=python#generate-explanations-for-predictions) and [schema](https://learn.microsoft.com/en-us/azure/machine-learning/reference-automl-images-schema#data-format-for-online-scoring-and-explainability-xai) articles."
"For more details on explainability with AutoML for images, refer to the [generating explanations](https://learn.microsoft.com/en-us/azure/machine-learning/how-to-auto-train-image-models?tabs=python#generate-explanations-for-predictions) and [schema](https://learn.microsoft.com/en-us/azure/machine-learning/reference-automl-images-schema#data-format-for-online-scoring-and-explainability-xai) articles.\n",
"\n",
"**Note**: Please note that explainability is not supported for Hugging Face models."
]
},
{
Expand Down
Loading

0 comments on commit 4df5a43

Please sign in to comment.