devops cicd
Azure DevOps Pipeline CI pipeline fundamentals
What is a CICD pipeline?
[if !supportLists]· [endif]In software engineering, CI/CD or CICD generally refers to the combined practices of continuous integration and either continuous delivery or continuous deployment.
Continuous integration (CI) is the practice used by development teams to simplify the testing and building of code. CI helps to catch bugs or problems early in the development cycle, which makes them easier and faster to fix.
Continuous delivery (CD) is a process by which code is built, tested, and deployed to one or more test and production stages. Deploying and testing in multiple stages helps drive quality.
[if !supportLists]· [endif]CI/CD bridges the gaps between development and operation activities and teams by enforcing automation in building, testing, and deployment of applications.
[if !supportLists]· [endif]A CI/CD pipeline for Machine Learning model lifecycle can be like below:
Feature set
A CICD pipeline can consist of multiple components. To implement our CICD pipeline, we used Azure Pipelines as part of Azure DevOps Services.
[if !supportLists]· [endif]A pipeline defines the continuous integration and deployment process for your app. It can be thought of as a workflow that defines how your test, build, and deployment steps are run.
[if !supportLists]· [endif]Stages, jobs, and steps are the main building blocks of a Pipeline. A stage can consist of one or more jobs, and a job can consist of one or more steps. A step is the smallest building block of a pipeline to perform an action, that can be a task or script.
[if !supportLists]· [endif]When a pipeline gets executed is controlled by the Trigger. A pipeline can be configured to run upon a push to a repository, at scheduled times, or upon the completion of another build. All of these actions are known as triggers.
[if !supportLists]· [endif]To start the CICD pipeline, at least one agent is required. An agent is a computing infrastructure with installed agent software that runs one job at a time. Azure DevOps provides Microsoft-hosted agents that waives the setup and maintenance effort for users, self-hosted agent is also an option when Microsoft-hosted agents do not meet requirements.
Now we know the key concepts of a CICD pipeline and how it works on Azure DevOps, we can start building the first CICD pipeline on Azure DevOps.
Azure CICD Pipeline Tutorial
There are multiple ways of setting up the CICD pipeline on Azure DevOps. In this introduction, we will start from a simple one via YAML definition.
Azure Pipelines doesn’t support all YAML features. Unsupported features include anchors, complex keys, and sets. Also, unlike standard YAML, Azure Pipelines depends on seeing stage, job, task, or a task shortcut like script as the first key in a mapping.
Basic Requirements
[if !supportLists]· [endif]An Azure DevOps account
[if !supportLists]· [endif]Git
[if !supportLists]· [endif]YAML
Initial Setup
[if !supportLists]· [endif]Create an empty repo named TemplateRepo
[if !supportLists]· [endif]Inside the repo, create an empty YAML file named azure-pipeline.yml and copy-paste the following lines into that file.
[if !supportLists]· [endif]Commit this file to the remote repo on master branch.
[if !supportLists]· [endif]name: the pipeline name in a string as cicd_ci
[if !supportLists]· [endif]trigger: branch name master will trigger a build
[if !supportLists]· [endif]stages: there is one stage named as CI_Checks , within this stage, there is one job with a friendly given name yamllint_checks . The job runs by the Microsoft-hosted agent uses VM image ubuntu-latest
[if !supportLists]· [endif]script: a shortcut for command-line task, the script consists of 3 commands as listed, and the command will be running in order. The command can be replaced with other working commands.
Now, we can create a new pipeline using this YAML definition. Go to your DevOps, on the left panel, Pipelines -> Pipelines -> New pipeline (a button on top right corner), then you will enter the page below. Depends on where your YAML file is located, you can choose where to import it:
In our case, we pick Azure Repos Git and then select the repository name and YAML definition.
Once the import is done, you can run your first pipeline. The pipeline run page provides a summary that contains information relevant to this pipeline run. Because we set the pipeline trigger as branch as master, which means the changes to the master branch will automatically start the pipeline run.
The running status of our defined stage CI_Checksand job yamllint_checks are showed here. You may notice that except our defined step YAML Lint Checks , there are multiple steps we did not specify, for example, Checkout xxx these are initialized and tear-down steps defined by default.
Complex pipeline
We will briefly cover the complex pipeline and execution orders as well.
So far we defined a simple pipeline, with one stage, one job, and one step. A pipeline, in reality, can consist of multiple stages, jobs, and steps. For example, deployment to different environments can be defined as multiple stages, within each stage multiple jobs are defined, demo as the following structure: