This is one chapter of a multi-part tutorial where we discuss how to integrate Azure DevOps with AWS.
In this tutorial we will learn how to integrate AWS CloudFormation with Azure DevOps pipelines.
- AWS CloudFormation Introduction
- How to manually deploy a CloudFormation template
- How to automate CloudFormation deployments with PowerShell scripts
- How to integrate Azure DevOps pipelines with AWS CloudFormation using Azure DevOps marketplace plugins
- How to integrate Azure DevOps pipelines with AWS CloudFormation using PowerShell
This is one chapter of a series of posts where we discuss how to integrate Azure DevOps with AWS.
AWS CloudFormation introduction
AWS Cloud Formation provides an easy way to write infrastructure as code. You can define which AWS resources to render and with what settings. You can also provision third-party components. If you are not familiar with AWS CloudFormation watch the following video to quickly learn the basics of AWS CloudFormation. If you are already familiar with AWS CloudFormation, you can safely skip this introduction.
How to manually deploy an AWS CloudFormation template
Once you are familiar with AWS CloudFormation, let’s do a manual deployment of a simple AWS CloudFormation template. The template is written in json, but you can also write CloudFormation templates in yaml format. The json formatted template in our experiment is given below. Since it provisions a simple VPC and a subnet, it will not cost you anything in your AWS account.
{
"AWSTemplateFormatVersion" : "2010-09-09",
"Resources" : {
"myVPC" : {
"Type" : "AWS::EC2::VPC",
"Properties" : {
"CidrBlock" : "10.0.0.0/16",
"EnableDnsSupport" : "false",
"EnableDnsHostnames" : "false",
"InstanceTenancy" : "default" ,
"Tags": [
{
"Key": "Name",
"Value": {
"Ref": "AWS::StackName"
}
}
]
}
},
"Sub1":{
"Type" : "AWS::EC2::Subnet",
"Properties" : {
"AvailabilityZone": {
"Fn::Select": [
"0",
{ "Fn::GetAZs": "" }
]
},
"CidrBlock" : "10.0.1.0/24",
"VpcId" : {"Ref":"myVPC" }
}
}
},
"Outputs":{
"SubnetIAID": {
"Description": "Subnet 1A ID in Availability Zone 1",
"Value": {
"Ref": "Sub1"
},
"Export": {
"Name": {
"Fn::Sub": "${AWS::StackName}-PrivateSubnet1AID"
}
}
}
}
}
Follow the instructions below to manually deploy the above CloudFormation template. Note that by default, when you are manually deploying the stack, the permission of the current user is used to deploy the AWS resources. You can also use a pre-defined IAM role so that CloudFormation engine can assume that role to deploy AWS resources. Make sure you have enough permissions to deploy a simple VPC and a subnet.
When you are done with your stack creation, delete the stack and observe how all the resources CloudFormation creates, get deleted along with the stack deletion.
How to automate CloudFormation deployments with PowerShell scripts
Manual deployments of CloudFormation is good for ad-hoc deployments. In production CICD pipelines, you want automated scripts to execute the CloudFormation templates. Most CICD systems like Azure DevOps, Jenkins, TravisCI, Bamboo or TeamCity support command line and PowerShell execution. In this scenario, you can add a command line or a PowerShell component into your CICD pipeline and configure it to execute AWS CLI commands or AWS tools for PowerShell cmdlets to deploy CloudFormation templates. Let’s first deploy a CloudFormation template with the help of PowerShell.
[Video timestamp:0min-10min 10sec]: We first try to execute the CloudFormation template without proper permissions. It will give an access denied error and fail. We then attach an IAM policy called MyCatApp-Policy to DevMachine’s IAM role and try to execute the template again. If you missed my previous post, remember to read more about AWS tools for PowerShell credential search order and EC2 instance profile. MyCatApp-Policy is given below. It gives just enough permission to deploy the resources we want. Granting the least set of permissions to your CICD pipeline is always a security best practice. It limits the damage a wrongly configured pipeline can do to the production environment.
MyCatApp-Policy. Note that this is a CloudFormation template to deploy the MyCatApp-Policy. If you plan to copy-paste this to create an IAM policy, you need to replace dynamic parameters like { “Ref”: “AWS::Region” } with proper AWS region values such as “ap-southeast-2”.
{
"AWSTemplateFormatVersion": "2010-09-09",
"Resources": {
"MyCatAppPolicy": {
"Type": "AWS::IAM::ManagedPolicy",
"Properties": {
"Description": "Sample policy showing how to grant fine grain permissions for MyCatApplication",
"ManagedPolicyName": "MyCatApp-Policy",
"Path": "/",
"PolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:*"
],
"Resource": "*",
"Condition": {
"StringEquals": {
"aws:RequestedRegion": { "Ref": "AWS::Region" }
}
}
},
{
"Effect": "Allow",
"Action": [
"cloudformation:Describe*",
"cloudformation:EstimateTemplateCost",
"cloudformation:Get*",
"cloudformation:List*",
"cloudformation:ValidateTemplate"
],
"Resource": "*",
"Condition": {
"StringEquals": {
"aws:RequestedRegion": { "Ref": "AWS::Region" }
}
}
},
{
"Effect": "Allow",
"Action": [
"cloudformation:CreateStack",
"cloudformation:DeleteStack",
"cloudformation:UpdateStack"
],
"Resource": "arn:*:cloudformation:*:*:stack/CatHome/*",
"Condition": {
"StringEquals": {
"aws:RequestedRegion": { "Ref": "AWS::Region" }
}
}
},
{
"Effect": "Allow",
"Action": [
"ec2:createTags",
"ec2:CreateVpc",
"ec2:DeleteVpc",
"ec2:DescribeVpcAttribute",
"ec2:DescribeVpcs",
"ec2:ModifyVpcAttribute",
"ec2:AssociateVpcCidrBlock",
"ec2:DisassociateVpcCidrBlock",
"ec2:DescribeAccountAttributes",
"ec2:DescribeAvailabilityZones",
"ec2:DescribeSubnets",
"ec2:CreateSubnet",
"ec2:DeleteSubnet",
"ec2:AssociateSubnetCidrBlock",
"ec2:DisassociateSubnetCidrBlock",
"ec2:ModifySubnetAttribute"
],
"Resource": "*",
"Condition": {
"StringEquals": {
"aws:RequestedRegion": { "Ref": "AWS::Region" }
}
}
}
]
}
}
}
}
}[Video timestamp:14min 10sec-16min 15sec]: Read MyCatApp-Policy carefully. If you change the PowerShell parameter $stackName to something like CatHome1 you will get an access denied exception. Can you think of using similar permissions to isolate different environments?
How to integrate Azure DevOps pipelines with AWS CloudFormation using Azure DevOps marketplace plugins
There are AWS provided Azure DevOps marketplace plugins you can add to your pipeline. Although they are not the most flexible compared to scripting everything using PowerShell, they give you a nice user interface to quickly create pipelines.
It’s crucial that you don’t specify any connection in the plugin’s AWS Credentials (Leave it blank). This will allow the plugin to fall back into credentials associated with E2 instance profile of the Azure DevOps agent. If you specify a connection, that connection gets the priority.
[Video timestamp: 12min 20sec – 19min 45sec] The plugins also allows you to specify IAM access keys and secret keys.Make sure the user has the permissions associated with MyCatApp-Policy. If you specify a connection to AWS, that connection get prioratized instead of EC2 instance role related permissions.
[Video timestamp:29min 30sec – 36min] You can use Azure repo to maintain your CloudFormation templates. This will allow you to have a versioned infrastructure. Make sure you have enabled continuous integration for the pipeline and observe how your CloudFormation gets deployed from the time you make modification to your CloudFormation template and commit it.
How to integrate Azure DevOps pipelines with AWS CloudFormation using PowerShell
We already learned how to execute a CloudFormation template using PowerShell. We will expand on that learnings and try to get our PowerShell script working on Azure DevOps pipeline. This allows us to do more advance customization to our pipeline.
This is one chapter of a multi-part tutorial where we discuss how to integrate Azure DevOps with AWS. For the next chapter visit Azure DevOps with AWS CodeDeploy.
Pingback: Azure DevOps with AWS - Cloudopian
Pingback: AWS PowerShell tutorial - Cloudopian