Azure DevOps with AWS CodeDeploy

In this tutorial, you will learn how to integrate AWS CodeDeploy with Azure DevOps to variety of deployments including hybrid deployments. We will build and deploy a simple console application into AWS EC2 instances and to on-premises environment in a single Azure DevOps pipeline. The on-premises environment is simulated with a laptop. We will discuss following topics and it’s better you follow the topics in the given order.

1. AWS CodeDeploy introduction
2. Introduction into what we will be building in this tutorial
3. How to create an Azure DevOps pipeline to build an AWS CodeDeploy deployment package
4. How to upload AWS CodeDeploy deployment package to Amazon S3 with Azure DevOps pipeline
5. How to create an AWS CodeDeploy application
6. Integrating Azure DevOps pipeline with AWS CodeDeploy
7. Hybrid deployments with AWS CodeDeploy and Azure DevOps

This is one chapter of a series of posts where we discuss how to integrate Azure DevOps with AWS.

AWS CodeDeploy introduction

AWS CodeDeploy is an agent based deployment tool. You can use it to deploy solutions into AWS, Azure, Google Cloud, On-premises or any other environment where you can install an agent. The tool itself is free and both Linux and Windows operating systems are supported. If you are not familiar with AWS CodeDeploy quickly watch the following introductory video.

Introduction into what we will be building in this tutorial

We will execute a series of steps to integrate AWS CodeDeploy with Azure DevOps. In the following video you will get a high-level overview of the Azure DevOps pipeline will be building in this tutorial. At the end of this tutorial you will be able to do hybrid deployments.

How to create an Azure DevOps pipeline to build an AWS CodeDeploy deployment package

We now have a good understanding of AWS CodeDeploy, its anatomy and what we will be building in this tutorial. Let’s now get our hands dirty and build the first part of Azure DevOps pipeline. Before doing any deployments, we first need to get our deployment package ready. Deployment package is zip file with a specific file structure. In the root directory of the zip file you have a configuration file called AppSpec which defines what to deploy, how to deploy and what events to trigger. Make sure your deployment package is correct before moving into the next steps.

Quickly read about the predefined variables available for you in Azure DevOps pipelines.

How to upload AWS CodeDeploy deployment package to Amazon S3 with Azure DevOps pipeline

Our Azure DevOps pipeline can now create the AWS CodeDeploy zip package. We also verified that the package is in the right format. Our next task is to get this zip file from Azure DevOps build agent into Amazon S3 so that we can point AWS CodeDeploy to pick the deployment artifacts from Amazon S3.  

How to create an AWS CodeDeploy application

By now our Azure DevOps pipeline can create an AWS CodeDeploy zip package and upload it to Amazon S3. We still haven’t touch anything related to AWS CodeDeploy in AWS console. The next step is to create an AWS CodeDeploy application. AWS CodeDeploy application is an abstraction to organize an application specific deployments. An application has a version, a set of environments that you can deploy it into and a history of deployments. Watch the following video to learn how to create an AWS CodeDeploy application.

• [Video timestamp: 0min-6min] Learn how to create an AWS CodeDeploy application
• [Video timestamp: 6min-12min 30sec] Deploy our simple console app (MyDonkeyApp) to DevMachine. Observe how the PowerShell scripts get executed.
• [Video timestamp: 12min 30sec – 15min] You need to have AWS CodeDeploy agent setup on your target machine. Verify that AWS CodeDeploy agent is running as a service in the target EC2 instance. AWS CodeDeploy agent generates a log file. This can be a great way to find any issues related to the agent. A common issues such as access denied errors can be identified with this log file.    
• [Video timestamp: 15min – 19min] We add an additional machine (BuildMachine) to the deployment group and do a new deployment targeting two machines.

Integrating Azure DevOps pipeline with AWS CodeDeploy

Make sure your build agent or the Azure DevOps pipeline has right permission to upload the content into Amazon S3 and also to trigger AWS CodeDeploy deploymWe are almost done. The next step is to put the final pieces together so that immediately after our pipeline uploads the deployment zip package to Amazon S3, it triggers the AWS CodeDeploy to make a deployment. Make sure your build agent or the Azure DevOps pipeline has right permission to upload the content into Amazon S3 and also to trigger AWS CodeDeploy deployment.

Hybrid deployments with AWS CodeDeploy and Azure DevOps

The real power of AWS CodeDeploy comes from its ability to do hybrid deployments. Please note that this tutorial requires you to install AWS CodeDeploy agent on your laptop or desktop to simulate the on-premises environment. Alternatively, you can install this agent in an Azure or Google Cloud VM. This part of the tutorial is quite lengthy. You can use the video timestamps given at the end to quickly jumps into different steps.

• [Video timestamp:0min-4min] Learn the high-level steps we will execute to setup the hybrid deployment environment.
• [Video timestamp:4min-5min 18sec] As shown in the video, install AWS tools for PowerShell. You can download AWS tools for PowerShell from https://aws.amazon.com/powershell/
• [Video timestamp:5min 18sec-8min 20sec] How to install AWS CodeDeploy agent. You can download the agent from following location. https://docs.aws.amazon.com/codedeploy/latest/userguide/codedeploy-agent-operations-install-windows.html . Make sure the AWS CodeDeploy agent service starts properly
• [Video timestamp:8min 20sec – 10min 6sec] How to grant access rights to AWS CodeDeploy agent so that it can read the deployment zip package from the Amazon S3 bucket. Note that the AWS CodeDeploy agent directly reads the deployment package from the Amazon S3 bucket instead of getting the zip file from some sort of an AWS CodeDeploy API endpoint.  To do this we create an IAM user called my-onpremises-machine-user. AWS CodeDeploy agent will then use this user’s access key and secret key to download the deployment packages. Since this is a quick test, we attach the permission AmazonS3ReadOnlyAccess to this IAM user. Ideally in production environments, we give fine-grained permissions that only allows access to specific bucket/folder/file path.
• [Video timestamp: 10min 6sec-11min 6sec] We also create an IAM user called my-codedeploy-onprem-reg-user. We will use this user’s access key and secret key to register the on-premises machine. We attach the permission AWSCodeDeployFullAccess to this user so that it can register the on-premises instance against the correct deployment group. As said before, ideally you should grant fine grained permissions to this user so that you can only invoke registration related commands against a specific application and a deployment group. Unlike in this lab environment, in production environments, we give fine-grained permissions. Once the instance is registered, you can remove this user/keys.Remember to note down the access key and secret key you just created.     
• [Video timestamp: 11min 6sec-18min 55sec] We alter conf.onpremises.yml file, add the credentials of my-onpremises-machine-user and restart the AWS CodeDeploy agent. We then verify the logs of AWS CodeDeploy agent to make sure it does not give any access denied errors and it can access Amazon S3 bucket where we keep deployment artifacts.
• [Video timestamp: 18min 55sec-22min 35sec] We then open PowerShell, set the credentials of my-codedeploy-onprem-reg-user s’ credentials on PowerShell session and then register the on-premises instance/laptop using Register-CDOnPremiseInstance cmdlet. Command documentation is given at https://docs.aws.amazon.com/powershell/latest/reference/items/Register-CDOnPremiseInstance.html . We then verify that the on-premises instance is properly registered under AWS CodeDeploy.
• [Video timestamp: 22min 35sec-26min] We still haven’t categorized our newly registered instance into a deployment group. We provide a tag to identify which on-premises instances belong into which deployment groups. We then create a new hybrid deployment group, and include DevMachine and our newly registered on-premises instance/laptop in this deployment group.
• [Video timestamp: 26min – 28min 30sec] We manually execute a new deployment to deploy MyDonkeyApp to on-premises instance and EC2 based DevMachine. We then modify the Azure DevOps pipeline in MyDonkeyApp to do an end-to-end delivery to hybrid deployment group.
• [Video timestamp: 28min 30sec – 28min 30sec] Its recommended you do a bit of cleaning before finishing this tutorial. Remember to deregister the code deployment agent (use the PowerShell cmdlet  Unregister-CDOnPremiseInstance), delete both my-onpremises-machine-user and  my-codedeploy-onprem-reg-user. Uninstall and reset any artifacts you may have installed (as part of this tutorial) in your laptop/on-premises instance.     

This is one chapter of a multi-part tutorial where we discuss how to integrate Azure DevOps with AWS. For the next chapter visit Azure DevOps with AWS Elastic Beanstalk.