Setup MLOPS workspace using Azure DevOps pipeline

Machine Learning is training machines using set of data to do some valuable predictions for the different business and social needs. So, data scientists put lot of effort to train more usable and valuable mathematical representation of real world process which is known as a model. To do the training process it is very important to have a good computational resources to train the model. Azure Machine Learning workspace is one of the well-known model training and building platform many data scientists use. If data scientist and DevOps engineers work together, they can automate the model training and deployment process using CI/CD pipelines. This post discusses, how to create a Machine Learning (ML) workspace in Azure by using a Azure DevOps build, which will allow you to use for training and testing a model.

Pre requisites: Azure subscription and an Azure DevOps Account

Go to Azure DevOps project pipelines. Create a new pipeline using classic editor.

Most of the time ML scripts are written in python. So,we select Ubuntu as the build agent to run builds faster and efficiently. Hence, we use shell scripts and bash scripts while configuring the build pipelines.

In this post Azure CLI commands available with Azure ML extension to Azure CLI are used to create the ML workspace using the Azure DevOps pipeline. Before start the ML workspace creation, it is required to install Azure CLI ML extension to the build agent machine. To do that add Shell script to the pipeline with following inline command.

az extension add -n azure-cli-ml

Add another Azure CLI task to the pipeline and change script type as shell. Select the script location as Inline and give following command to create the ML workspace in Azure. You need to provide workspace name, resource group name, location to create the workspace, sku (store keeping unit) to create the workspace. As enhancement to the command add --yes part which check for the resource group availability and if there isn't any resource group with the provided name, create new resource group. Also, --exist-ok part of the command is to check whether there is workspace available with the same name if not it creates a new ML workspace.

az ml workspace create --resource-group rg-mldemodev01 --workspace-name mlw-demodev01  --location southcentralus --sku enterprise --yes --exist-ok

Add another Azure CLI task as shell script to the pipeline. ML workspace is using to train the ML models. So, it needs computational power. Next, command is to create a Azure ML cluster inside the workspace. It will create a cluster with minimum 0 VMs and maximum 3 VMs with given vm size inside the ML workspace.

az ml computetarget create amlcompute --resource-group rg-mldemodev01 --workspace-name mlw-demodev01 --min-nodes 0 --max-nodes 3 --name cmbamlcompute01 --vm-size STANDARD_DS2_V2 --idle-seconds-before-scaledown 300

Workspace is not completed without data. So, next step is to upload the data to data store in Azure. We can do that using the Azure CLI command. Add Azure CLI command as shell script and use following script to upload data to data store. It selects the default data store created for the workspace and upload the data files to Azure data store. Source path specifies the data as the relative folder path which contains the data files.

az ml datastore upload --resource-group rg-mldemodev01 --workspace-name mlw-demodev01 --name $(az ml datastore show-default --resource-group rg-mldemodev01 --workspace-name mlw-demodev01 --query name --output tsv) --src-path data --target-path diabetes --overwrite true

Now trigger a build and it will create a work space and cluster in your Azure subscription.

Such ML work space created in Azure can be used to train a model and run experiments.