diff --git a/README.md b/README.md index 64ab49bc..324f3249 100644 --- a/README.md +++ b/README.md @@ -11,36 +11,33 @@ description: "Code which demonstrates how to set up and operationalize an MLOps # MLOps with Azure ML - [![Build Status](https://aidemos.visualstudio.com/MLOps/_apis/build/status/microsoft.MLOpsPython?branchName=master)](https://aidemos.visualstudio.com/MLOps/_build/latest?definitionId=151&branchName=master) - -MLOps will help you to understand how to build the Continuous Integration and Continuous Delivery pipeline for a ML/AI project. We will be using the Azure DevOps Project for build and release/deployment pipelines along with Azure ML services for model retraining pipeline, model management and operationalization. +MLOps will help you to understand how to build the Continuous Integration and Continuous Delivery pipeline for a ML/AI project. We will be using the Azure DevOps Project for build and release/deployment pipelines along with Azure ML services for model retraining pipeline, model management and operationalization. ![ML lifecycle](/docs/images/ml-lifecycle.png) This template contains code and pipeline definition for a machine learning project demonstrating how to automate an end to end ML/AI workflow. The build pipelines include DevOps tasks for data sanity test, unit test, model training on different compute targets, model version management, model evaluation/model selection, model deployment as realtime web service, staged deployment to QA/prod and integration testing. - ## Prerequisite + - Active Azure subscription - At least contributor access to Azure subscription -## Getting Started: +## Getting Started To deploy this solution in your subscription, follow the manual instructions in the [getting started](docs/getting_started.md) doc - ## Architecture Diagram -This reference architecture shows how to implement continuous integration (CI), continuous delivery (CD), and retraining pipeline for an AI application using Azure DevOps and Azure Machine Learning. The solution is built on the scikit-learn diabetes dataset but can be easily adapted for any AI scenario and other popular build systems such as Jenkins and Travis. +This reference architecture shows how to implement continuous integration (CI), continuous delivery (CD), and retraining pipeline for an AI application using Azure DevOps and Azure Machine Learning. The solution is built on the scikit-learn diabetes dataset but can be easily adapted for any AI scenario and other popular build systems such as Jenkins and Travis. ![Architecture](/docs/images/main-flow.png) - ## Architecture Flow ### Train Model + 1. Data Scientist writes/updates the code and push it to git repo. This triggers the Azure DevOps build pipeline (continuous integration). 2. Once the Azure DevOps build pipeline is triggered, it performs code quality checks, data sanity tests, unit tests, builds an [Azure ML Pipeline](https://docs.microsoft.com/en-us/azure/machine-learning/service/concept-ml-pipelines) and publishes it in an [Azure ML Service Workspace](https://docs.microsoft.com/en-us/azure/machine-learning/service/concept-azure-machine-learning-architecture#workspace). 3. The [Azure ML Pipeline](https://docs.microsoft.com/en-us/azure/machine-learning/service/concept-ml-pipelines) is triggered once the Azure DevOps build pipeline completes. All the tasks in this pipeline runs on Azure ML Compute. Following are the tasks in this pipeline: @@ -56,13 +53,13 @@ This reference architecture shows how to implement continuous integration (CI), Once you have registered your ML model, you can use Azure ML + Azure DevOps to deploy it. The [Azure DevOps multi-stage pipeline](https://docs.microsoft.com/en-us/azure/devops/pipelines/process/stages?view=azure-devops&tabs=yaml) packages the new model along with the scoring file and its python dependencies into a [docker image](https://docs.microsoft.com/en-us/azure/machine-learning/service/concept-azure-machine-learning-architecture#image) and pushes it to [Azure Container Registry](https://docs.microsoft.com/en-us/azure/container-registry/container-registry-intro). This image is used to deploy the model as [web service](https://docs.microsoft.com/en-us/azure/machine-learning/service/concept-azure-machine-learning-architecture#web-service) across QA and Prod environments. The QA environment is running on top of [Azure Container Instances (ACI)](https://azure.microsoft.com/en-us/services/container-instances/) and the Prod environment is built with [Azure Kubernetes Service (AKS)](https://docs.microsoft.com/en-us/azure/aks/intro-kubernetes). - ### Repo Details You can find the details of the code and scripts in the repository [here](/docs/code_description.md) ### References + - [Azure Machine Learning(Azure ML) Service Workspace](https://docs.microsoft.com/en-us/azure/machine-learning/service/overview-what-is-azure-ml) - [Azure ML CLI](https://docs.microsoft.com/en-us/azure/machine-learning/service/reference-azure-machine-learning-cli) - [Azure ML Samples](https://docs.microsoft.com/en-us/azure/machine-learning/service/samples-notebooks) @@ -73,7 +70,7 @@ You can find the details of the code and scripts in the repository [here](/docs/ This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us -the rights to use your contribution. For details, visit https://cla.microsoft.com. +the rights to use your contribution. For details, visit When you submit a pull request, a CLA-bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., label, comment). Simply follow the instructions diff --git a/bootstrap/README.md b/bootstrap/README.md new file mode 100644 index 00000000..bbbbff09 --- /dev/null +++ b/bootstrap/README.md @@ -0,0 +1,9 @@ +# Bootstrap from MLOpsPython repository + +To use this existing project structure and scripts for your new ML project, you can quickly get started from the existing repository, bootstrap and create a template that works for your ML project. Bootstraping will prepare a similar directory structure for your project which includes renaming files and folders, deleting and cleaning up some directories and fixing imports and absolute path based on your project name. This will enable reusing various resources like pre-built pipelines and scripts for your new project. + +To bootstrap from the existing MLOpsPython repository clone this repository and run bootstrap.py script as below + +>python bootstrap.py --d [dirpath] --n [projectname] + +Where [dirpath] is the absolute path to the root of your directory where MLOps repo is cloned and [projectname] is the name of your ML project diff --git a/bootstrap/bootstrap.py b/bootstrap/bootstrap.py new file mode 100644 index 00000000..d796f6df --- /dev/null +++ b/bootstrap/bootstrap.py @@ -0,0 +1,135 @@ +import os +import sys +import argparse +# from git import Repo + + +class Helper: + + def __init__(self, project_directory, project_name): + self._project_directory = project_directory + self._project_name = project_name + self._git_repo = "https://github.com/microsoft/MLOpsPython.git" + + @property + def project_directory(self): + return self._project_directory + + @property + def project_name(self): + return self._project_name + + @property + def git_repo(self): + return self._git_repo + + # def clonerepo(self): + # # Download MLOpsPython repo from git + # Repo.clone_from( + # self._git_repo, self._project_directory, branch="master", depth=1) # NOQA: E501 + # print(self._project_directory) + + def renamefiles(self): + # Rename all files starting with diabetes_regression with project name + strtoreplace = "diabetes_regression" + dirs = [".pipelines", r"ml_service\pipelines"] + for dir in dirs: + dirpath = os.path.join(self._project_directory, dir) + for filename in os.listdir(dirpath): + if(filename.find(strtoreplace) != -1): + src = os.path.join(self._project_directory, dir, filename) + dst = os.path.join(self._project_directory, + dir, filename.replace(strtoreplace, self._project_name, 1)) # NOQA: E501 + os.rename(src, dst) + + def renamedir(self): + # Rename any directory with diabetes_regression with project name + dirs = ["diabetes_regression"] + for dir in dirs: + src = os.path.join(self._project_directory, dir) + dst = os.path.join(self._project_directory, self._project_name) + os.rename(src, dst) + + def deletedir(self): + # Delete unwanted directories + dirs = ["docs", r"diabetes_regression\training\R"] + for dir in dirs: + os.system( + 'rmdir /S /Q "{}"'.format(os.path.join(self._project_directory, dir))) # NOQA: E501 + + def replaceprojectname(self): + # Replace instances of diabetes_regression within files + dirs = [r".env.example", + r".pipelines\azdo-base-pipeline.yml", + r".pipelines\azdo-pr-build-train.yml", + r".pipelines\diabetes_regression-ci-build-train.yml", + r".pipelines\diabetes_regression-ci-image.yml", + r".pipelines\diabetes_regression-template-get-model-version.yml", # NOQA: E501 + r".pipelines\diabetes_regression-variables.yml", + r"environment_setup\Dockerfile", + r"environment_setup\install_requirements.sh", + r"ml_service\pipelines\diabetes_regression_build_train_pipeline_with_r_on_dbricks.py", # NOQA: E501 + r"ml_service\pipelines\diabetes_regression_build_train_pipeline_with_r.py", # NOQA: E501 + r"ml_service\pipelines\diabetes_regression_build_train_pipeline.py", # NOQA: E501 + r"ml_service\pipelines\diabetes_regression_verify_train_pipeline.py", # NOQA: E501 + r"ml_service\util\create_scoring_image.py", + r"diabetes_regression\azureml_environment.json", + r"diabetes_regression\conda_dependencies.yml", + r"diabetes_regression\evaluate\evaluate_model.py", + r"diabetes_regression\training\test_train.py"] # NOQA: E501 + + for file in dirs: + fin = open(os.path.join(self._project_directory, file), + "rt", encoding="utf8") + data = fin.read() + data = data.replace("diabetes_regression", self.project_name) + fin.close() + fin = open(os.path.join(self._project_directory, file), + "wt", encoding="utf8") + fin.write(data) + fin.close() + + def cleandir(self): + # Clean up directories + dirs = ["data", "experimentation"] + for dir in dirs: + for root, dirs, files in os.walk(os.path.join(self._project_directory, dir)): # NOQA: E501 + for file in files: + os.remove(os.path.join(root, file)) + + def validateargs(self): + # Validate arguments + if (os.path.isdir(self._project_directory) is False): + raise Exception( + "Not a valid directory. Please provide absolute directory path") # NOQA: E501 + # if (len(os.listdir(self._project_directory)) > 0): + # raise Exception("Directory not empty. PLease empty directory") + if(len(self._project_name) < 3 or len(self._project_name) > 15): + raise Exception("Project name should be 3 to 15 chars long") + + +def main(args): + parser = argparse.ArgumentParser(description='New Template') + parser.add_argument("--d", type=str, + help="Absolute path to new project direcory") + parser.add_argument( + "--n", type=str, help="Name of the project[3-15 chars] ") + try: + args = parser.parse_args() + project_directory = args.d + project_name = args.n + helper = Helper(project_directory, project_name) + helper.validateargs() + # helper.clonerepo() + helper.cleandir() + helper.replaceprojectname() + helper.deletedir() + helper.renamefiles() + helper.renamedir() + except Exception as e: + print(e) + return 0 + + +if '__main__' == __name__: + sys.exit(main(sys.argv)) diff --git a/docs/code_description.md b/docs/code_description.md index d60df616..1b0b710d 100644 --- a/docs/code_description.md +++ b/docs/code_description.md @@ -1,5 +1,34 @@ ## Repo Details +### Directory Structure + +High level directory structure for this repository: + +```bash +├── .pipelines <- Azure DevOps YAML pipelines for CI, PR and model training and deployment. +├── bootstrap <- Python script to initialize this repository with a custom project name. +├── charts <- Helm charts to deploy resources on Azure Kubernetes Service(AKS). +├── data <- Initial set of data to train and evaluate model. +├── diabetes_regression <- The top-level folder for the ML project. +│ ├── evaluate <- Python script to evaluate trained ML model. +│ ├── register <- Python script to register trained ML model with Azure Machine Learning Service. +│ ├── scoring <- Python score.py to deploy trained ML model. +│ ├── training <- Python script to train ML model. +│ ├── R <- R script to train R based ML model. +│ ├── util <- Python script for various utility operations specific to this ML project. +├── docs <- Extensive markdown documentation for entire project. +├── environment_setup <- The top-level folder for everything related to infrastructure. +│ ├── arm-templates <- Azure Resource Manager(ARM) templates to build infrastructure needed for this project. +├── experimentation <- Jupyter notebooks with ML experimentation code. +├── ml_service <- The top-level folder for all Azure Machine Learning resources. +│ ├── pipelines <- Python script that builds Azure Machine Learning pipelines. +│ ├── util <- Python script for various utility operations specific to Azure Machine Learning. +├── .env.example <- Example .env file with environment for local development experience. +├── .gitignore <- A gitignore file specifies intentionally un-tracked files that Git should ignore. +├── LICENSE <- License document for this project. +├── README.md <- The top-level README for developers using this project. +``` + ### Environment Setup - `environment_setup/install_requirements.sh` : This script prepares a local conda environment i.e. install the Azure ML SDK and the packages specified in environment definitions. @@ -8,7 +37,7 @@ - `environment_setup/Dockerfile` : Dockerfile of a build agent containing Python 3.6 and all required packages. -- `environment_setup/docker-image-pipeline.yml` : An AzDo pipeline for building and pushing [microsoft/mlopspython](https://hub.docker.com/_/microsoft-mlops-python) image. +- `environment_setup/docker-image-pipeline.yml` : An AzDo pipeline for building and pushing [microsoft/mlopspython](https://hub.docker.com/_/microsoft-mlops-python) image. ### Pipelines @@ -37,10 +66,11 @@ - `diabetes_regression/evaluate/evaluate_model.py` : an evaluating step of an ML training pipeline which registers a new trained model if evaluation shows the new model is more performant than the previous one. - `diabetes_regression/evaluate/register_model.py` : (LEGACY) registers a new trained model if evaluation shows the new model is more performant than the previous one. - `diabetes_regression/training/R/r_train.r` : training a model with R basing on a sample dataset (weight_data.csv). -- `diabetes_regression/training/R/train_with_r.py` : a python wrapper (ML Pipeline Step) invoking R training script on ML Compute +- `diabetes_regression/training/R/train_with_r.py` : a python wrapper (ML Pipeline Step) invoking R training script on ML Compute - `diabetes_regression/training/R/train_with_r_on_databricks.py` : a python wrapper (ML Pipeline Step) invoking R training script on Databricks Compute - `diabetes_regression/training/R/weight_data.csv` : a sample dataset used by R script (r_train.r) to train a model ### Scoring + - `diabetes_regression/scoring/score.py` : a scoring script which is about to be packed into a Docker Image along with a model while being deployed to QA/Prod environment. - `diabetes_regression/scoring/inference_config.yml`, deployment_config_aci.yml, deployment_config_aks.yml : configuration files for the [AML Model Deploy](https://marketplace.visualstudio.com/items?itemName=ms-air-aiagility.private-vss-services-azureml&ssr=false#overview) pipeline task for ACI and AKS deployment targets. diff --git a/docs/getting_started.md b/docs/getting_started.md index 009ae6c0..0b4b8379 100644 --- a/docs/getting_started.md +++ b/docs/getting_started.md @@ -8,8 +8,12 @@ following the instructions [here](https://docs.microsoft.com/en-us/azure/devops/ If you already have an Azure DevOps organization, create a [new project](https://docs.microsoft.com/en-us/azure/devops/organizations/projects/create-project?view=azure-devops). -## Clone or fork this repository -Fork this repository within GitHub, or clone it into your Azure DevOps project. +## Decide best option to copy repository code + +* Fork this repository if there is a desire to contribute back to the repository else +* Use this [code template](https://github.com/microsoft/MLOpsPython/generate) which copies the entire code base to your own GitHub location with the git commit history restarted. This can be used for learning and following the guide. + +If the desire is to use this project for your machine learning code, follow the [bootstrap instructions](../bootstrap/README.md) after the code template is complete. ## Create an ARM Service Connection to deploy resources @@ -48,16 +52,16 @@ Create a variable group named **``devopsforai-aml-vg``**. The YAML pipeline defi The variable group should contain the following required variables: -| Variable Name | Suggested Value | -| --------------------------- | -----------------------------------| -| BASE_NAME | [unique base name] | -| LOCATION | centralus | -| RESOURCE_GROUP | mlops-RG | -| WORKSPACE_NAME | mlops-AML-WS | -| WORKSPACE_SVC_CONNECTION | aml-workspace-connection | -| ACI_DEPLOYMENT_NAME | diabetes-aci | +| Variable Name | Suggested Value | +| ------------------------ | ------------------------ | +| BASE_NAME | [unique base name] | +| LOCATION | centralus | +| RESOURCE_GROUP | mlops-RG | +| WORKSPACE_NAME | mlops-AML-WS | +| WORKSPACE_SVC_CONNECTION | aml-workspace-connection | +| ACI_DEPLOYMENT_NAME | diabetes-aci | -**Note:** +**Note:** The **WORKSPACE_NAME** parameter is used for the Azure Machine Learning Workspace creation. You can provide an existing AML Workspace here if you have one. @@ -67,7 +71,7 @@ be naming collisions with resources that require unique names like azure blob storage and registry DNS naming. Make sure to give a unique value to the BASE_NAME variable (e.g. MyUniqueML), so that the created resources will have unique names (e.g. MyUniqueMLamlcr, MyUniqueML-AML-KV, etc.). The length of -the BASE_NAME value should not exceed 10 characters and it should contain numbers and letters only. +the BASE_NAME value should not exceed 10 characters and it should contain numbers and letters only. The **RESOURCE_GROUP** parameter is used as the name for the resource group that will hold the Azure resources for the solution. If providing an existing AML Workspace, set this value to the corresponding resource group name. @@ -121,11 +125,11 @@ Check out the newly created resources in the [Azure Portal](https://portal.azure (Optional) To remove the resources created for this project you can use the [/environment_setup/iac-remove-environment.yml](../environment_setup/iac-remove-environment.yml) definition or you can just delete the resource group in the [Azure Portal](https://portal.azure.com). -**Note:** The training ML pipeline uses a [sample diabetes dataset](https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_diabetes.html) as training data. If you want to use your own dataset, you need to [create and register a datastore](https://docs.microsoft.com/en-us/azure/machine-learning/how-to-access-data#azure-machine-learning-studio) in your ML workspace and upload the datafile (e.g. [diabetes.csv](./data/diabetes.csv)) to the corresponding blob container. You can also define a datastore in the ML Workspace with [az cli](https://docs.microsoft.com/en-us/cli/azure/ext/azure-cli-ml/ml/datastore?view=azure-cli-latest#ext-azure-cli-ml-az-ml-datastore-attach-blob). +**Note:** The training ML pipeline uses a [sample diabetes dataset](https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_diabetes.html) as training data. If you want to use your own dataset, you need to [create and register a datastore](https://docs.microsoft.com/en-us/azure/machine-learning/how-to-access-data#azure-machine-learning-studio) in your ML workspace and upload the datafile (e.g. [diabetes.csv](./data/diabetes.csv)) to the corresponding blob container. You can also define a datastore in the ML Workspace with [az cli](https://docs.microsoft.com/en-us/cli/azure/ext/azure-cli-ml/ml/datastore?view=azure-cli-latest#ext-azure-cli-ml-az-ml-datastore-attach-blob). You'll also need to configure DATASTORE_NAME and DATAFILE_NAME variables in ***devopsforai-aml-vg*** variable group. - ## Create an Azure DevOps Azure ML Workspace Service Connection + Install the **Azure Machine Learning** extension to your organization from the [marketplace](https://marketplace.visualstudio.com/items?itemName=ms-air-aiagility.vss-services-azureml), so that you can set up a service connection to your AML workspace. @@ -201,17 +205,17 @@ The final stage is to deploy the model to the production environment running on [Azure Kubernetes Service](https://azure.microsoft.com/en-us/services/kubernetes-service). **Note:** Creating a Kubernetes cluster on AKS is out of scope of this -tutorial, but you can find set up information +tutorial, but you can find set up information [here](https://docs.microsoft.com/en-us/azure/aks/kubernetes-walkthrough-portal#create-an-aks-cluster). **Note:** If your target deployment environment is a K8s cluster and you want to implement Canary and/or A/B testing deployment strategies check out this [tutorial](./canary_ab_deployment.md). In the Variables tab, edit your variable group (`devopsforai-aml-vg`). In the variable group definition, add the following variables: -| Variable Name | Suggested Value | -| --------------------------- | -----------------------------------| -| AKS_COMPUTE_NAME | aks | -| AKS_DEPLOYMENT_NAME | diabetes-aks | +| Variable Name | Suggested Value | +| ------------------- | --------------- | +| AKS_COMPUTE_NAME | aks | +| AKS_DEPLOYMENT_NAME | diabetes-aks | Set **AKS_COMPUTE_NAME** to the *Compute name* of the Inference Cluster referencing your AKS cluster in your Azure ML Workspace. @@ -226,9 +230,9 @@ scoring service on Azure App Service](https://docs.microsoft.com/en-us/azure/mac In the Variables tab, edit your variable group (`devopsforai-aml-vg`). In the variable group definition, add the following variable: -| Variable Name | Suggested Value | -| --------------------------- | -----------------------------------| -| WEBAPP_DEPLOYMENT_NAME | mlopswebapp | +| Variable Name | Suggested Value | +| ---------------------- | --------------- | +| WEBAPP_DEPLOYMENT_NAME | mlopswebapp | Set **WEBAPP_DEPLOYMENT_NAME** to the name of your Azure Web App. Delete the **ACI_DEPLOYMENT_NAME** variable.