AWS PowerShell tutorial

This is one chapter of a multi-part tutorial where we discuss how to integrate Azure DevOps with AWS.

AWS tools for PowerShell is essential to create advanced CICD pipelines that integrate with AWS. Whether you use Azure DevOps, Jenkins, Team City, Travis CI or Bamboo to create your CICD pipelines all of them support execution of PowerShell commands. All AWS APIs are exposed as PowerShell cmdlets and being able to integrate with AWS using PowerShell gives an immense design flexibility to your pipeline. Since there are lot of cmdlets, as a beginner, it may be overwhelming to decide where to start. Luckily for you, all the commands follow the same pattern.  When you learn a few commands, the rest of the commands are easy to understand.

AWS tools for PowerShell introduction

Watch the following AWS PowerShell introduction video where we walk you through the basics of AWS PowerShell cmdlets. All what you need to know is the credential search order and where to find help when you want to refer to an AWS API call.  

Basic credential provisioning

There are numerous ways to access AWS using PowerShell. In the following video tutorial, we will learn the most basic method to access AWS using IAM access key and secret key.

Remember, although this is the easiest method to access AWS, it’s not recommended to use IAM credentials in this way. These credentials are static and can leak through logs, monitoring systems or someone hacking into your pipeline. It’s always recommended to use temporary access keys and secret keys which will learn later in this tutorial.

You can also use the IAM user credentials in the PowerShell session. However, when you close the PowerShell window, those session credentials are lost and you need to re initiate them again. In the next section we will learn how to use credential profiles to securely store the credentials in user’s folder.

Credential Profiles

Credential profiles allow you to securely store named IAM access key and secret keys so that you can easily refer to the access key and secret key using a name. These credential profiles can be saved in any file location. However, those profiles maintained in the current user’s folder can only be accessed by the current user or the administrator. Watch the following video tutorial to understand how credential profiles work.

If you want to get your hands dirty and try how credential profile works, do the following exercises.  

• Create a profile in the default location to keep the access key and the secret key. Can someone other than the user who creates it access those credentials?
• How to get a list of profiles saved in the default profile location?
• Use credentials stored in a profile to list Amazon S3 buckets
• Create a profile, but this time keep it under a given file name. Use the credential in the file to list S3 buckets.
• Can you keep credential files in a network shared location? Think about some use cases why you want to maintain the credential file in a network shared location
• How many profiles can you keep per user per machine?
• Think how you can you use different profiles (with just enough permission granted) to do different deployment tasks

Whether you use the IAM access key and secret key directly in PowerShell or using a credential profile, the access key and the secret key remain static until you change it. This poses some security challenges since a compromised access key and a secret key can give access to the AWS environment until it is rotated, deactivated or deleted. In the next video we will explore a feature of Amazon EC2 called instance profile to grant access to PowerShell cmdlets.

EC2 instance profile

An instance profile associates a set of AWS IAM permissions with an EC2 instance. After the association, any program running on the EC2 instance will magically get the permission associated with that instance. You don’t need to invoke extra commands to get AWS access. As long as you are using AWS provided SDKs (such as AWS tools for PowerShell, AWS Command Line Interface (CLI) or any other language specific SDKs like AWS SDK for Java or .NET) your application will get AWS access without special credential initialization. Access keys and secret keys obtained through EC2 instance profile are temporary. Therefore, even if the keys get compromised, the attacker has limited time to use them. AWS security features like Amazon Guard Duty can detect usage of access keys and secret keys issued for EC2 instances outside EC2 instance IP address ranges and fire warnings.

Watch the following video to learn how to use EC2 instance profile to grant access to PowerShell. This approach is useful when you want to grant AWS permissions to your Azure DevOps agent running on EC2.

AWS Security Token Service (STS)

What if your Azure DevOps agents are not in Amazon EC2 and you still want to have temporary credentials to access AWS? This is common if your Azure DevOps agents are on Azure or on on-premises servers. In this situation, you can use a service called AWS Security Token Service (STS) to get temporarily What if your Azure DevOps agents are not in Amazon EC2 but you still want to have temporary credentials to access AWS? This is common if your Azure DevOps agents are on Azure or on on-premises servers. In this situation, you can use a service called AWS Security Token Service (STS) to get temporary credentials. AWS STS is the underpinning technology behind all the authentication & authorization mechanisms on AWS. Whether you use SAML based authentication such as ADFS, OpenID connect, AWS CLI or SDK to access AWS, all of them use AWS STS to obtain temporary credentials. Therefore, having a deep understanding of AWS STS is crucial step towards mastering AWS security. Watch the following video to learn how AWS Security token service works. In this video we use AWS PowerShell, nevertheless the API is available in Command Line Interface (CLI), .NET, Java, Python or any other language AWS supports.

[Video timestamp: 0min-3min]: We first remove AmazonS3FullAccess from the EC2 instance role. This will ensure when we access S3 from PowerShell, credentials are not coming from the EC2 instance role. We then create a new user called mytest-user with no permissions and create an access key and a secret key. The important thing to note here is that mytest-user does not have any permission. We will later use this user and assume a role called my-test-role which has some permissions.  

[Video timestamp:3min-6min]: We create a new role called my-test-role and attached the permission AmazonS3FullAccess to it. Modify the trust section of the role to trust the user mytest-user (we created a moment ago) to assume the role. This will ensure not all users can assume the role, but only the given user can assume the role. We configure the trust relationship to include following configuration. Although we configure an IAM user as the trusted principal, you can also define another role or a service as the trusted principal. For instance, if you want another role X to assume my-test-role you will then define the urn of X as the trusted principal.

 
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::123456789123:user/mytest-user"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

Note that the AWS account number 123456789123 will be different for you.

The flexibility to configure this trust relationship is enormous. A role can be configured/trusted to assume another role in another account.  This is useful when you want to promote a deployment in the pre-production account into production. We can configure the trust relationship, such that a user in a shared service account/pre-production account is trusted to assume a role in production environment to do production deployments.

[Video timestamp: 6min-8min 30sec:] We then use the cmdlet Use-STSRole to assume the role my-test-role and get temporary credentials.

[Video timestamp:8min 30sec-10min 10sec] The temporary credentials you got from Use-STSRole can then be used to list S3 buckets. Note that unlike static credentials associated with an IAM user, these new temporary access key and secret key pair has an expiration time.  Also observe how the credential changes every time you call Use-STSRole cmdlet.  

[Video timestamp:10min 10sec-13min] There is a minimum duration of 900 seconds or 15min you can assume a role.  In a large distributed system with millions of users having these kind of limits allows to manage the sheer volume of API requests that can easily go into trillions. If you try to use the credentials after they have been expired, you will get an error message saying that the credentials are no longer valid. If you use .NET, Java or any other AWS provided SDKs, trying to use expired credentials will throw an exception which you can later catch and use to assume the role again.

[Video timestap:13min-15min] With the help of new assume role approach, you can now have a shared service that can deploy solutions to multiple environments/AWS accounts and none of those destinations need to maintain static access keys or secret keys or even IAM users.   All what those destinations need is a role which trusts a shared service IAM user or a role.

This is one chapter of a multi-part tutorial where we discuss how to integrate Azure DevOps with AWS. For the next chapter visit Azure DevOps with AWS CloudFormation.