Azure DevOps with AWS Fargate Engine

AWS Fargate is a serverless compute engine for containers that works with both Amazon Elastic Container Service (ECS) and Amazon Elastic Kubernetes Service (EKS). Fargate makes it easy for you to focus on building your applications. Fargate removes the need to provision and manage servers, lets you pay for resources per application, and improves security through application isolation by design. It does not have the complexity of Kubernetes but provides the same scalability. In this tutorial we will learn how to integrate AWS Fargate with Azure DevOps pipelines. We will cover following topics.

1. Container introduction
2. AWS container orchestration technologies
3. Initial lab setup and Azure DevOps pipeline configuration for AWS Fargate
4. How to publish the container image to Amazon Elastic Container Registry (ECR) with Azure DevOps
5. How to create an AWS Fargate task definition
6. How to create an AWS Fargate cluster
7. How to create an AWS Fargate service
8. Integrate Azure DevOps pipeline with AWS Fargate to do and end-to-end delivery

This is one chapter of a multi-part tutorial where we discuss how to integrate Azure DevOps with AWS.

Container introduction

If you are completely new to containers quickly go through following container introductory video. If you are already familiar with containers you can safely skip this video.

AWS container orchestration technologies

According to a recent study, more than 63% of container workloads run on AWS and more than 51% of Kubernetes workload runs on AWS. AWS has a variety of platforms to run your container workloads. This include Amazon Elastic Container Service (ECS), Amazon Elastic Container Registry (ECR), Amazon Elastic Kubernetes Service (EKS) and AWS Fargate to name a few. AWS Fargate by far is the easiest and fastest way to get your containers up and running with minimal management and operational overheads. In the following video, you will learn various container orchestration platforms AWS provides.

Initial lab setup and Azure DevOps pipeline configuration for AWS Fargate

We first need to have an application to deploy into AWS Fargate. In the following video, we will create a simple ASP.NET core web application and create a docker image out of it. The source code for the application is available at https://github.com/cloudopian/azure-devops-on-aws/tree/master/MyWolfApp

• [Video timestamp: 0sec – 4min 50sec] We first build the MyWolfApp website in DevMachine and test it. We then make a modification to the home page and commit it to Azure Repo. Since we can’t build a Linux container image on a Windows machine (at least at the time of this writing), we will build the image using an Linux Azure DevOps agent.
• [Video timestamp: 4min 50sec – 12min 45sec] We start the Linux build agent so that we can build Linux container image. Once the build agent comes alive, we execute the pipeline and test whether the agent can build the container image.

How to publish the container image to Amazon Elastic Container Registry (ECR) with Azure DevOps

In the previous videos we got our container image built. But it is still inside our Azure DevOps agent. The next step is to publish this image into Amazon ECR so that we can point AWS Fargate to it. With AWS Fargate, you are not limited to use Amazon ECR. You can host your container images in any other Docker repository such as docker hub and Azure Container Registry and point AWS Fargate to take the image from them. However, Amazon ECR provides the best fine-grained security integration features compared to other repositories. The level of individual API calls you can control is amazing.  Watch the following video to understand how to publish a Docker container image from Azure DevOps pipeline.  

Note that throughout this tutorial, when granting permissions, we follow an easy approach (E.g. attaching the managed permission AmazonEC2ContainerRegistryFullAccess). In production environments, you must create fine grained permissions like the one below that gives access only to a specific ECR repository (E.g. mywolfapp).

{
"Effect": "Allow",
"Action": [
           "ecr:*"
            ],
"Resource": "arn:aws:ecr:ap-southeast-2:123456789101:repository/mywolfapp",
"Condition":{
               "StringEquals": {
                  "aws:RequestedRegion": "ap-southeast-2"
                }
             }
} 

How to create an AWS Fargate task definition

In the previous video we got our container image published into Amazon ECR. The next step is to create an AWS Fargate task so that we can point Fargate cluster to your image. If you are coming from a Kubernetes background a task is more or less like a Kubernetes Pod. A task maintains parameters like

• Which Docker image to run
• What ports to open
• How much memory and CPU to be allocated to the container
• How to mount volumes for storage
• Where to keep log files

Watch the following tutorial to create an AWS Fargate task.

How to create an AWS Fargate cluster

In the In the previous video, we created an AWS Fargate task. We now need to have a Fargate cluster to run the task. A cluster provides resources like network, memory, cpu and storage. Since AWS Fargate is a fully managed environment, creating a cluster is super easy. Watch the following video to learn how to create an AWS Fargate cluster and test it.

How to create an AWS Fargate service

You can run individual tasks on a cluster. However, what will happen if the tasks fail? A failure can happen due a software bug or due to an AWS instance failure. We need a mechanism to recover from such failures and start the tasks again. What if you want to load balance a workload between multiple tasks, say between two tasks that run the same container image. This is where a service comes handy. A service allows you to load balance between tasks and it can recover from a task failure. Service (either an Amazon ECS or AWS Fargate service) allows you to maintain a desired set of running tasks. When the service finds that the desired state is different from the running state, the service takes necessary actions to bring the running state to the desired state. The service is also the interface between load balancer and the running containers. It also allows you to gradually roll-out new deployments. This is useful when running micro services.Watch the following video to learn how to create an AWS Fargate service.

• [Video timestamp: 0sec – 1min 30sec]:  We create an AWS Fargate cluster.
• [Video timestamp:1min 30sec-6min 29sec]: We run the task we created in a previous section and test whether the cluster is working as expected.
• [Video timestamp:6min 10sec-6min 29sec]: We stop the running task

Integrate Azure DevOps pipeline with AWS Fargate to do and end-to-end delivery

If you reached this far, well done! We now have a working AWS Fargate cluster and a working AWS service running our ASP.Net core application. The final step is to configure our Azure DevOps pipeline to refresh the AWS service immediately after publishing a new container image into Amazon ECR. Since we have created our tasks in a way that it always points to the image with :latest tag, when you refresh the service, the newly published image will get deployed to the cluster. Watch the following video to understand how this is done.   

• [Video timestamp: 0min – 5min]: We explore the pipeline and understand how the end-to-end configuration has been done
• [Video timestamp: 5min – 9min 30sec]: We modify the build tasks to tag images with correct repository URL and point the publication of the images to the correct repository.  We then attach the right permission to Azure DevOps agent so that it can refresh the ECS service, and finally execute the pipeline.
• [Video timestamp: 9min 30sec-11min 50sec]: We observe how the container deployment happens behind the scene.
• [Video timestamp: 9min 30sec-18min]: To finally test the workings of end-to-end pipeline, we make some modification to the website code and commit it to the Azure repository and observe how the end-to-end integration works.

This is one chapter of a multi-part tutorial where we discuss how to integrate Azure DevOps with AWS.