Download code from GitHub
Amazon Web Services (AWS) is a public cloud provider. It provides infrastructure and platform services at a pay-per-use rate. This means you get on-demand access to resources that you used to have to buy outright. You can get access to enterprise-grade services while only paying for what you need, usually down to the hour.
AWS prides itself on providing the primitives to developers so that they can build and scale the solutions that they require.
In order to follow along with the recipes, you will need an AWS account. Create an account at https://aws.amazon.com/ by clicking on the Sign Up button and entering your details.
Note
Even though we will be taking advantage of the free tier wherever possible, you will need a valid credit card to complete the signup process. Go to https://aws.amazon.com/free/ for more information. Note that the free tier only applies for the first year of your account's lifetime.
A fundamental concept of AWS is that its services and the solutions built on top of them are architected for failure. This means that a failure of the underlying resources is a scenario actively planned for, rather than avoided until it cannot be ignored.
Due to this, all the services and resources available are divided up in to geographically diverse Regions. Using specific regions means you can provide services to your users that are optimized for speed and performance.
Within a region, there are always multiple Availability Zones (a.k.a. AZ). Each AZ represents a geographically distinct—but still close—physical data center. AZs have their own facilities and power source, so an event that might take a single AZ offline is unlikely to affect the other AZs in the region.
The smaller regions have at least two AZs, and the largest has five.
At the time of writing, the following regions are active:
Code | Name | Availability Zones |
| N. Virginia | 5 |
| Ohio | 3 |
| N. California | 3 |
| Oregon | 3 |
| Canada | 2 |
| Ireland | 3 |
| London | 2 |
| Frankfurt | 2 |
| Tokyo | 3 |
| Seoul | 2 |
| Singapore | 2 |
| Sydney | 3 |
| Mumbai | 2 |
| Sao Paulo | 3 |
The web-based console is the first thing you will see after creating your AWS account, and you will often refer to it when viewing and confirming your configuration.
The AWS web console
The console provides an overview of all the services available as well as associated billing and cost information. Each service has its own section, and the information displayed depends on the service being viewed. As new features and services are released, the console will change and improve. Don't be surprised if you log in and things have changed from one day to the next.
Note
Keep in mind that the console always shows your resources by region. If you cannot see a resource that you created, make sure you have the right region selected.
Choose the region closest to your physical location for the fastest response times. Note that not all regions have the same services available. The larger, older regions generally have the most services available. Some of the newer or smaller regions (that might be closest to you) might not have all services enabled yet. While services are continually being released to regions, you may have to use another region if you simply must use a newer service.
Note
The us-east-1 (a.k.a. North Virginia) region is special given its status as the first region. All services are available there, and new services are always released there.
As you get more advanced with your use of AWS, you will spend less time in the console and more time controlling your services programmatically via the AWS CLI tool and CloudFormation, which we will go into in more detail in the next few topics.
Where possible, we have based the recipes around a CloudFormation template. CloudFormation is the Infrastructure as Code service from AWS.
Note
Where CloudFormation was not applicable, we have used the AWS CLI to make the process repeatable and automatable.
Since the recipes are based on CloudFormation templates, you can easily combine different templates to achieve your desired outcomes. By editing the templates or joining them, you can create more useful and customized configurations with minimal effort.
Infrastructure as Code (IaC) is the practice of managing infrastructure though code definitions.
On an Infrastructure-as-a-Service (IaaS) platform such as AWS, IaC is needed to get the most utility and value. IaC differs primarily from traditional interactive methods of managing infrastructure because it is machine processable. This enables a number of benefits:
- Improved visibility of resources
- Higher levels of consistency between deployments and environments
- Easier troubleshooting of issues
- The ability to scale more with less effort
- Better control over costs
On a less tangible level, all of these factors contribute to other improvements for your developers: you can now leverage tried-and-tested software development practices for your infrastructure and enable DevOps practices in your teams.
As your infrastructure is represented in machine-readable files, you can treat it like you do your application code. You can take the best-practice approaches to software development and apply them to your infrastructure. This means you can store it in version control (for example, Git and SVN) just like you do your code, along with the benefits that it brings:
- All changes to infrastructure are recorded in commit history
- You can review changes before accepting/merging them
- You can easily compare different configurations
- You can pick and use specific point-in-time configurations
Consistent configuration across your environments (for example, dev, test, and prod) means that you can more confidently deploy your infrastructure. When you know what configuration is in use, you can easily test changes in other environments due to a common baseline.
IaC is not the same as just writing scripts for your infrastructure. Most tools and services will leverage higher-order languages and DSLs to allow you to focus on your higher-level requirements. It enables you to use advanced software development techniques, such as static analysis, automated testing, and optimization.
IaC makes replicating and troubleshooting issues easier: since you can duplicate your environments, you can accurately reproduce your production environment for testing purposes.
In the past, test environments rarely had exactly the same infrastructure due to the prohibitive cost of hardware. Now that it can be created and destroyed on demand, you are able to duplicate your environments only when they are needed. You only need to pay for the time that they are running for, usually down to the hour. Once you have finished testing, simply turn your environments off and stop paying for them.
Even better than troubleshooting is fixing issues before they cause errors. As you refine your IaC in multiple environments, you will gain confidence that is difficult to obtain without it. By the time you deploy your infrastructure in to production, you have done it multiple times already.
Configuring infrastructure by hand can be a tedious and error-prone process. By automating it, you remove the potential variability of a manual implementation: computers are good at boring, repetitive tasks, so use them for it!
Once automated, the labor cost of provisioning more resources is effectively zero—you have already done the work. Whether you need to spin up one server or a thousand, it requires no additional work.
From a practical perspective, resources in AWS are effectively unconstrained. If you are willing to pay for it, AWS will let you use it.
AWS have a vested (commercial) interest in making it as easy as possible for you to provision infrastructure. The benefit to you as the customer is that you can create and destroy these resources on demand.
Obviously, destroying infrastructure on-demand in a traditional, physical hardware environment is simply not possible. You would be hard-pressed to find a data center that will allow you to stop paying for servers and space simply because you are not currently using them.
Another use case where on-demand infrastructure can make large cost savings is your development environment. It only makes sense to have a development environment while you have developers to use it. When your developers go home at the end of the day, you can switch off your development environments so that you no longer pay for them. Before your developers come in in the morning, simply schedule their environments to be created.
DevOps and IaC go hand in hand. The practice of storing your infrastructure (traditionally the concern of Operations) as code (traditionally the concern of Development) encourages a sharing of responsibilities that facilitates collaboration.
Image courtesy: Wikipedia
By automating the PACKAGE, RELEASE, and CONFIGURE activities in the software development life cycle (as pictured), you increase the speed of your releases while also increasing confidence.
Cloud-based IaC encourages architecture for failure: as your resources are virtualized, you must plan for the chance of physical (host) hardware failure, however unlikely.
Being able to recreate your entire environment in minutes is the ultimate recovery solution.
Unlike physical hardware, you can easily simulate and test failure in your software architecture by deleting key components—they are all virtual anyway!
Server-side examples of IaC are configuration-management tools such as Ansible, Chef, and Puppet.
While important, these configuration-management tools are not specific to AWS, so we will not be covering them in detail here. There are a myriad of books and courses devoted to this topic if you need to know more.
CloudFormation is the IaC service from AWS.
Templates written in a specific format and language define the AWS resources that should be provisioned. CloudFormation is declarative and cannot only provision resources, but also update them.
We will go into CloudFormation in greater detail in the next topic.
We'll use CloudFormation extensively throughout this book, so it's important that you have an understanding of what it is and how it fits in to the AWS ecosystem. There should easily be enough information here to get you started, but where necessary, we'll refer you to AWS' own documentation.
The CloudFormation service allows you to provision and manage a collection of AWS resources in an automated and repeatable fashion. In AWS terminology, these collections are referred to as stacks. Note however that a stack can be as large or as small as you like. It might consist of a single S3 bucket, or it might contain everything needed to host your three-tier web app.
In this chapter, we'll show you how to define the resources to be included in your CloudFormation stack. We'll talk a bit more about the composition of these stacks and why and when it's preferable to divvy up resources between a number of stacks. Finally, we'll share a few of the tips and tricks we've learned over years of building countless CloudFormation stacks.
By now, the benefits of automation should be starting to become apparent to you. But don't fall in to the trap of thinking CloudFormation will be useful only for large collections of resources. Even performing the simplest task of, say, creating an S3 bucket can get very repetitive if you need to do it in every region.
We work with a lot of customers who have very tight controls and governance around their infrastructure, and especially in the network layer (think VPCs, NACLs, and security groups). Being able to express one's cloud footprint in YAML (or JSON), store it in a source code repository, and funnel it through a high-visibility pipeline gives these customers confidence that their infrastructure changes are peer-reviewed and will work as expected in production. Discipline and commitment to IaC SDLC practices are of course a big factor in this, but CloudFormation helps bring us out of the era of following 20-page run-sheets for manual changes, navigating untracked or unexplained configuration drift, and unexpected downtime caused by fat fingers.
Now is a good time to start thinking about your AWS deployments in terms of layers. Your layers will sit atop one another, and you will have well-defined relationships between them.
Here's a bottom-up example of how your layer cake might look:
- VPC with CloudTrail
- Subnets, routes, and NACLs
- NAT gateways, VPN or bastion hosts, and associated security groups
- App stack 1: security groups, S3 buckets
- App stack 1: cross-zone RDS and read replica
- App stack 1: app and web server auto scaling groups and ELBs
- App stack 1: CloudFront and WAF config
In this example, you may have many occurrences of the app stack layers inside your VPC, assuming you have enough IP addresses in your subnets! This is often the case with VPCs living inside development environments. So immediately, you have the benefit of multi-tenancy capability with application isolation.
One advantage of this approach is that while you are developing your CloudFormation template, if you mess up the configuration of your app server, you don't have to wind back all the work CFN did on your behalf. You can just turf that particular layer (and the layers that depend on it) and restart from there. This is not the case if you have everything contained in a single template.
We commonly work with customers for whom ownership and management of each layer in the cake reflects the structure of the technology divisions within a company. The traditional infrastructure, network, and cyber security folk are often really interested in creating a safe place for digital teams to deploy their apps, so they like to heavily govern the foundational layers of the cake. Conway's Law, coined by Melvin Conway, starts to come in to play here:
"Any organization that designs a system will inevitably produce a design whose structure is a copy of the organization's communication structure."
Finally, even if you are a single-person infrastructure coder working in a small team, you will benefit from this approach. For example, you'll find that it dramatically reduces your exposure to things such as AWS limits, timeouts, and circular dependencies.
This is where we start to get our hands dirty. CloudFormation template files are the codified representation of your stack, expressed in either YAML or JSON. When you wish to create a CloudFormation stack, you push this template file to CloudFormation, through its API, web console, command line tools, or some other method (such as the SDK).
Templates can be replayed over and over again by CloudFormation, creating many instances of your stack.
Up until recently, JSON was your only option. We'll actually encourage you to adopt YAML, and we'll be using it for all of the examples shown in this book. Some of the reasons are as follows:
- It's just nicer to look at. It's less syntax heavy, and should you choose to go down the path of generating your CloudFormation templates, pretty much every language has a YAML library of some kind.
- The size of your templates will be much smaller. This is more practical from a developer's point of view, but it also means you're less likely to run into the CloudFormation size limit on template files (50 KB).
- The string-substitution features are easier to use and interpret.
- Your EC2
UserData(the script that runs when your EC2 instance boots) will be much easier to implement and maintain.
CloudFormation templates consist of a number of parts, but these are the four we're going to concentrate on:
- Parameters
- Resources
- Outputs
- Mappings
Here's a short YAML example:
AWSTemplateFormatVersion: '2010-09-09'
Parameters:
EC2KeyName:
Type: String
Description: EC2 Key Pair to launch with
Mappings:
RegionMap:
us-east-1:
AMIID: ami-9be6f38c
ap-southeast-2:
AMIID: ami-28cff44b
Resources:
ExampleEC2Instance:
Type: AWS::EC2::Instance
Properties:
InstanceType: t2.nano
UserData:
Fn::Base64:
Fn::Sub': |
#!/bin/bash -ex
/opt/aws/bin/cfn-signal '${ExampleWaitHandle}'
ImageId:
Fn::FindInMap: [ RegionMap, Ref: 'AWS::Region', AMIID ]
KeyName:
Ref: EC2KeyName
ExampleWaitHandle:
Type: AWS::CloudFormation::WaitConditionHandle
Properties:
ExampleWaitCondition:
Type: AWS::CloudFormation::WaitCondition
DependsOn: ExampleEC2Instance
Properties:
Handle:
Ref: ExampleWaitHandle
Timeout: 600
Outputs:
ExampleOutput:
Value:
Fn::GetAtt: ExampleWaitCondition.Data
Description: The data signaled with the WaitConditionCloudFormation parameters are the input values you define when creating or updating your stack, similar to how you provide parameters to any command-line tools you might use. They allow you to customize your stack without making changes to your template. Common examples of what parameters might be used for are as follows:
- EC2 AMI ID: You may wish to redeploy your stack with a new AMI that has the latest security patches installed.
- Subnet IDs: You could have a list of subnets that an auto scaling group should deploy servers in. These subnet IDs will be different between your dev, test, and production environments.
- Endpoint targets and credentials: These include things such as API hostnames, usernames, and passwords.
You'll find that there are a number of parameter types. In brief, they are:
- String
- Number
- List
- CommaDelimitedList
In addition to these, AWS provides some AWS-specific parameter types. These can be particularly handy when you are executing your template via the CloudFormation web console. For example, a parameter type of AWS::EC2::AvailabilityZone::Name will cause the web console to display a drop-down list of valid Availability Zones for this parameter. In the ap-southeast-2 region, the list would look like this:
ap-southeast-2aap-southeast-2bap-southeast-2c
The list of AWS-specific parameter types is steadily growing and is large enough that we can't list them here. We'll use many of them throughout this book, however, and they can easily be found in the AWS CloudFormation documentation.
When creating or updating a stack, you will need to provide values for all the parameters you've defined in your template. Where it makes sense, you can define default values for a parameter. For example, you might have a parameter called debug that tells your application to run in debug mode. You typically don't want this mode enabled by default, so you can set the default value for this parameter to false, disabled, or something else your application understands. Of course, this value can be overridden when creating or updating your stack.
You can and should provide a short, meaningful description for each parameter. These are displayed in the web console next to each parameter field. When used properly, they provide hints and context to whoever is trying to run your CloudFormation template.
At this point, we need to introduce the inbuilt Ref function. When you need to reference a parameter value, you use this function to do so:
KeyName: Ref: EC2KeyName
While Ref isn't the only inbuilt function you'll need to know, it's almost certainly going to be the one you'll use the most. We'll talk more about inbuilt functions later in this chapter.
Resources are your actual pieces of AWS infrastructure. These are your EC2 instances, S3 buckets, ELBs, and so on. Almost any resource type you can create by pointing and clicking in the AWS web console can also be created using CloudFormation.
Note
It's not practical to list all the AWS resource types in this chapter, although you will get familiar with the most common types as you work your way through the recipes in this book. AWS keeps a definitive list of resources types here http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-template-resource-type-ref.html.
There are a few important things to keep in mind about CloudFormation resources:
- New or bleeding-edge AWS resources are often not immediately supported. CloudFormation support typically lags a few weeks (sometimes months) behind the release of new AWS features. This used to be quite frustrating for anyone to whom infrastructure automation is key. Fast-forward to today, and this situation is somewhat mitigated by the ability to use custom resources. These are discussed further on in this chapter.
- Resources have a default return value. You can use
Refto fetch these return values for use elsewhere in your template. For example, theAWS::EC2::VPCresource type has a default return value that is the ID of the VPC. They look something like this:vpc-11aa111a. - Resources often contain additional return values. These additional values are fetched using the inbuilt
Fn::GetAttfunction. Continuing from the previous example, theAWS::EC2::VPCresource type also returns the following:CidrBlockDefaultNetworkAclDefaultSecurityGroupIpv6CidrBlocks
Just like AWS resources, CloudFormation stacks can also have return values, called outputs. These values are entirely user defined. If you don't specify any outputs, then nothing is returned when your stack is completed.
Outputs can come in handy when you are using a CI/CD tool to create your CloudFormation stacks. For example, you might like to output the public hostname of an ELB so your CI/CD tool can turn it into a clickable link within the job output.
You'll also use them when your are linking together pieces of your layer cake. You may want to reference an S3 bucket or security group created in another stack. This is much easier to do with the new cross-stack references feature, which we'll discuss later in this chapter. You can expect to see the Ref and Fn::GetAtt functions a lot in the output section of any CloudFormation template.
The mappings section is used to define a set of key/value pairs. If you require any kind of AWS region portability, perhaps for DR or availability purposes or simply to get your application closer to your end user, you'll almost certainly need to specify some mappings in your template. This is particularly necessary if you are referencing anything in your template that is region specific.
The canonical example would be to specify a map of EC2 AMI IDs in your template. This is because AMIs are a region-specific resource, so a reference to a valid Amazon Machine Image (AMI) ID in one region will be invalid in another.
Mappings look like this:
Mappings: RegionMap: us-east-1: AMIID: ami-9be6f38c ap-southeast-2: AMIID: ami-28cff44b
When executing your template, CloudFormation will automatically work out which resources depend on each other and order their creation accordingly. Additionally, resource creation is parallelized as much as possible so that your stack execution finishes in the timeliest manner possible. Things occasionally become unstuck, however.
Let's take an example where an app server depends on a DB server. In order to connect to the database, the app server needs to know its IP address or hostname. This situation would actually require you to create the DB server first so that you can use Ref to fetch its IP and provide it to your app server. CloudFormation has no way of knowing about the coupling between these two resources, so it will go ahead and create them in any order it pleases (or in parallel if possible).
To fix this situation, we use the DependsOn attribute to tell CloudFormation that our app server depends on our DB server. In fact, DependsOn can actually take a list of strings if a resource happens to depend on multiple resources before it can be created. So if our app server were to also depend on, say, a Memcached server, then we use DependsOn to declare both dependencies.
If necessary, you can take this further. Let's say that after your DB server boots, it will automatically start the database, set up a schema, and import a large amount of data. It may be necessary to wait for this process to complete before we create an app server that attempts to connect to a DB expecting a complete schema and data set. In this scenario, we want a way to signal to CloudFormation that the DB server has completed its initialization so it can go ahead and create resources that depend on it. This is where WaitCondition and WaitConditionHandle come in.
Firstly, you create an AWS::CloudFormation::WaitConditionHandle type, which you can later reference via Ref.
Next, you create an AWS::CloudFormation::WaitCondition type. In our case, we want the wait period to start as soon as the DB server is created, so we specify that this WaitCondition resource DependsOn our DB server.
After the DB server has finished importing data and is ready to accept connections, it calls the callback URL provided by the WaitConditionHandle resource to signal to CloudFormation that it can stop waiting and start executing the rest of the CloudFormation stack. The URL is supplied to the DB server via UserData, again using Ref. Typically, curl, wget or some equivalent is used to call the URL.
A WaitCondition resource can have a Timeout period too. This is a value specified in seconds. In our example, we might supply a value of 900 because we know that it should never take more than 15 minutes to boot our DB and import the data.
Here's an example of what DependsOn, WaitConditionHandle, and WaitCondition look like combined:
ExampleWaitHandle: Type: AWS::CloudFormation::WaitConditionHandle Properties: ExampleWaitCondition: Type: AWS::CloudFormation::WaitCondition DependsOn: ExampleEC2Instance Properties: Handle: Ref: ExampleWaitHandle Timeout: 600
CloudFormation provides some inbuilt functions to make composing your templates a lot easier. We've already looked at Ref and Fn::GetAtt. Let's look at some others you are likely to encounter.
Use Fn::Join to concatenate a list of strings using a specified delimiter, like this, for example:
"Fn::Join": [ ".", [ 1, 2, 3, 4 ] ]
This would yield the following value:
"1.2.3.4"Use Fn::Sub to perform string substitution. Consider this:
DSN: "Fn::Sub"
- mysql://${db_user}:${db_pass}@${db_host}:3306/wordpress
- { db_user: lchan, db_pass: ch33s3, db_host: localhost }This would yield the following value:
mysql://lchan:ch33s3@localhost:3306/wordpressWhen you combine these functions with Ref and Fn::GetAtt, you can start doing some really powerful stuff, as we'll be seeing in the recipes throughout this book.
Other available inbuilt functions include:
Fn::Base64Fn::FindInMapFn::GetAZsFn::ImportValueFn::Select
Note
Documentation on all of these functions is available here http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/intrinsic-function-reference.html.
It's reasonably common to provision a similar but distinct set of resources based on which environment your stack is running in. In your development environment, for example, you may not wish to create an entire fleet of database servers (HA master and read slaves), instead opting for just a single database server. You can achieve this by using conditionals:
Fn::AndFn::EqualsFn::IfFn::NotFn::Or
One important thing to remember about CloudFormation is that it's more or less just making API calls on your behalf. This means that CloudFormation will assume the very same permissions or role you use to execute your template. If you don't have permission to create a new hosted zone in Route 53, for example, any template you try to run that contains a new Route 53-hosted zone will fail.
On the flip side, this has created a somewhat tricky situation where anyone developing CloudFormation typically has a very elevated level of privileges, and these privileges are somewhat unnecessarily granted to CloudFormation each time a template is executed.
If my CloudFormation template contains only one resource, which is a Route 53-hosted zone, it doesn't make sense for that template to be executed with full admin privileges to my AWS account. It makes much more sense to give CloudFormation a very slim set of permissions to execute the template with, thus limiting the blast radius if a bad template were to be executed (that is, a bad copy-and-paste operation resulting in deleted resources).
Thankfully, service roles have recently been introduced, and you can now define an IAM role and tell CloudFormation to use this role when your stack is being executed, giving you a much safer space to play in.
As discussed previously in this chapter, it's common for there to be a lengthy wait between the release of a new AWS feature and your ability to use that feature in CloudFormation.
Before custom resources, this led AWS developers down the path of doing over 95 percent of their automation in CloudFormation and then running some CLI commands to fill in the gaps. It was often difficult to tell exactly which resources belonged to which stack, and knowing exactly when your stack had finished execution became a guessing game.
Fast forward to today, and the emerging pattern is to use a custom resource to delegate to a AWS Lambda function. Lambda can fill in the gaps by making API calls on your behalf, and it becomes much easier to track the heritage and completion of these resources.
Note
With any luck, you won't need to use this feature for a while. In the meantime, the AWS custom resource documentation is quite comprehensive. If you are trying to use CloudFormation to create a resource that you can't find in the AWS docs, then it's likely that it's not supported in CloudFormation yet and using custom resources is your answer. For more information, refer to http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/template-custom-resources.html.
When using the layered cake approach, it's very common to want to use outputs from one stack as inputs in another stack. For example, you may create a VPC in one stack and require its VPC ID when creating resources in another.
For a long time, one needed to provide some glue around stack creation to pass output between stacks. AWS recently introduced cross-stack references, which provide a more native way of doing this.
You can now export one or more outputs from your stack. This makes those outputs available to other stacks. Note that the name of this value needs to be unique, so it's probably a good idea to include the CloudFormation stack name in the name you're exporting to achieve this.
Once a value is exported, it becomes available to be imported in another stack using the Fn::ImportValue function—very handy!
Note
Make sure, however, that during the time an exported value is being referenced, you are not able to delete or modify it. Additionally, you won't be able to delete the stack containing the exported value. Once something is referencing an exported value, it's there to stay until there are no stacks referencing it at all.
One of the principles of IaC is that all changes should be represented as code for review and testing. This is especially important where CloudFormation is concerned.
After creating a stack for you, the CloudFormation service is effectively hands off. If you make a change to any of the resources created by CloudFormation (in the web console, command line, or by some other method), you're effectively causing configuration drift. CloudFormation no longer knows the exact state of the resources in your stack.
The correct approach is to make these changes in your CloudFormation template and perform an update operation on your stack. This ensures that CloudFormation always knows the state of your stack and allows you to maintain confidence that your infrastructure code is a complete and accurate representation of your running environments.
When performing a stack update, it can be unclear exactly what changes are going to be made to your stack. Depending on which resource you are changing, you may find that it will need to be deleted and recreated in order to implement your change. This, of course, is completely undesired behavior if the resource in question contains data you'd like to keep. Keep in mind that RDS databases can be a particular pain point.
To mitigate this situation, CloudFormation allows you to create and review a change set prior to executing the update. The change set shows you which operations CloudFormation intends to perform on your resources. If the change set looks good, you can choose to proceed. If you don't like what you see, you can delete the change set and choose another course of action—perhaps choosing to create and switch to an entirely new stack to avoid a service outage.
There are a few other things you should keep in the back of your mind as you start to build out your own CloudFormation stacks. Let's take a look.
Often, if you omit the name attribute from a resource, CloudFormation will generate a name for you. This can result in weird-looking resource names, but it will increase the replayability of your template. Using AWS::S3::Bucket as an example, if you specify the BucketName parameter but don't ensure its uniqueness, CloudFormation will fail to execute your template the second time around because the bucket will already exist. Omitting BucketName fixes this. Alternatively, you may opt to generate your own unique name each time the template is run. There's probably no right or wrong approach here, so just do what works for you.
When creating a CloudFormation stack, you are given the option of disabling rollback. Before you go ahead and set this to true, keep in mind that this setting persists beyond stack creation. We've ended up in precarious situations where updating an existing stack has failed (for some reason) but rollback has been disabled. This is a fun situation for no one.
The limits most likely to concern you are as follows:
- The maximum size allowed for your CloudFormation template is 50 KB. This is quite generous, and if you hit this limit, you almost certainly need to think about breaking up your template into a series of smaller ones. If you absolutely need to exceed the 50 KB limit, then the most common approach is to first upload your template to S3 and then provide an S3 URL to CloudFormation to execute.
- The maximum number of parameters you can specify is 60. If you need more than this then again, consider whether or not you need to add more layers to your cake. Otherwise, lists or mappings might get you out of trouble here.
- Outputs are also limited to 60. If you've hit this limit, it's probably time to resort to a series of smaller templates.
- Resources are limited to 200. The same rules apply here as before.
- By default, you're limited to a total of 200 CloudFormation stacks. You can have this limit increased simply by contacting AWS.
Something to keep in the back of your mind is that you may run in to a circular dependency scenario, where multiple resources depend on each other for creation. A common example is where two security groups reference each other in order to allow access between themselves.
A workaround for this particular scenario is to use the AWS::EC2::SecurityGroupEgress and AWS::EC2::SecurityGroupIngress types instead of the ingress and egress rule types for AWS::EC2::SecurityGroup.
DSLs and generators can be a point of hot debate among infrastructure coders. Some love them, some hate them. Some of the reasons why people love them include the following:
- They allow CloudFormation to be written in a language that is more native to them or their team.
- They allow the use some advanced programming constructs. Iteration is a particularly well-cited example.
- Until YAML was supported by CloudFormation, using a DSL usually resulted in code that was easier to read and far less verbose.
Some of the reasons people dislike them are:
- DSLs have a history of becoming abandonware or significantly lagging behind CloudFormation, although there are a couple of well-supported DSLs out there
- Developers are potentially required to learn a new language and navigate another new set of documentation, on top of learning CloudFormation and navigating the AWS documentation
- Google and Stack Overflow become a little less useful because one needs to translate questions and answers
Beyond what is written here, this topic won't come up again in this book. We can't give specific advice as to which road you should take because it's almost always a highly personal and situational choice. However, a sensible approach, especially while coming to grips with AWS and CloudFormation, would be to stick with YAML (or JSON) until you get to the point where you think a DSL or generator might be useful.
Under no circumstances do you want to have credentials hardcoded in your templates or committed to your source code repository. Doing this doesn't just increase the chance your credentials will be stolen, it also reduces the portability of your templates. If your credentials are hardcoded and you need to change them, that obviously requires you to edit your CloudFormation template.
Instead, you should add credentials as parameters in your template. Be sure to use the NoEcho parameter when you do this so that CloudFormation masks the value anywhere the parameters are displayed.
If there are resources in your stack you'd like to protect from accidental deletion or modification, applying a stack policy will help you achieve this. By default, all resources are able to be deleted or modified. When you apply a stack policy, all resources are protected unless you explicitly allow them to be deleted or modified in the policy. Note that stack policies do not apply during stack creation—they only take effect when you attempt to update a stack.
The AWS command-line interface (CLI) tool is an important piece of the AWS administrator's toolkit.
The CLI tool is often one of the quickest and easiest ways to interact with the API. As a text-based tool, it scales much easier than using the web console. Unlike the console, it can be automated, for example, via scripts. The AWS application programming interface (API) represents all the functionality available to you as an AWS administrator. It is also easier to keep a track of through your command-line history. Like all good CLI tools, simple individual commands can be chained (or piped) together to perform complex tasks.
Note
The CLI tool is open source software, maintained on GitHub https://github.com/aws/aws-cli. For more detailed documentation, refer to the AWS CLI homepage https://aws.amazon.com/cli.
The CLI tool requires Python 2.6.5 or greater.
The easiest way to install it is to use the Python package manager, pip:
pip install awscliThis will make the command aws available on your system.
AWS frequently releases new services and functionality. In order to use the new features, you will need to upgrade the CLI tool.
To upgrade, run the following pip command periodically:
pip install --upgrade awscliAuthentication between the CLI tool and the AWS API is done via two pieces of information:
- Access key ID
- Secret access key
Note
As the name suggests, you should keep your secret access key a secret! Be careful where you store or send it.
Once you have created a user, you can configure the tool to use it for authentication purposes.
While you can configure the CLI tool with access keys directly, this should be avoided. Instead, you should use profiles to store your credentials. Using profiles gives you a more consistent and manageable centralized location to secure your secret keys.
Without any additional configuration or options, your CLI tool commands will use the default profile.
To set up the default profile, you can use the following command:
aws configureThis will prompt you for an access key ID, secret access key, region, and output format.
In addition to the default profile, you can configure other, named profiles. This is useful for switching between users with different levels of access (for example, read-only and administrator) or even between users in different accounts.
aws configure --profile <profile-name>Once you have responded to the prompts, you can reference the named profile by passing the --profile <profile-name> option with your command.
You can also configure the CLI via the use of environment variables:
export AWS_PROFILE=<profile-name>While you should prefer to use profiles over setting your access ID and secret keys directly, sometimes you may have to do it. If you must set your keys directly, do so via environment variables so that you do not need to pass your keys around or hardcode them:
export AWS_ACCESS_KEY_ID=<access-key-id> export AWS_SECRET_ACCESS_KEY=<secret-access-key>
When running the CLI tool on an EC2 instance, you can leverage the instance's IAM role to make calls. This means you do not need to configure credentials or set environment variables (manually).
Behind the scenes, the instance will retrieve and set its own AWS environment variables that allow API calls. You do need to ensure the instance has appropriate permissions.
All CLI tool commands are service based. Using service commands and subcommands, you can make calls directly to the AWS API.
Each command represents an AWS service. While most services have one command associated with them, some services have multiple commands (for example, S3 has s3 and s3api).
Note
Run aws help to see all the commands/services that are available—they will probably have changed by the time this book prints!
Each command has a selection of subcommands to perform service-specific actions.
Subcommands take options, which start with --.
While most are optional (hence the name), those that are not surrounded by square brackets ([]) are required. You will get an error message (with appropriate details) if you do not include them.
The built-in documentation is the best place to start looking for answers. There are usually examples after all of the options have been described. Otherwise, there are plenty of examples available online.
Some options are available to all or most commands, so they are particularly useful to know.
The CLI tool can be configured to output in JSON, table, or text format. To control the output type, use the --output option.
To set a default output type for all your commands, set the output parameter for your profile.
JavaScript Object Notation (JSON) (http://json.org/), a standard, machine- and human-readable information interchange format. Here's what the AZs in the us-east-1 (North Virginia) region look like represented as JSON:
aws ec2 describe-availability-zones --output json
{
"AvailabilityZones": [
{
"State": "available",
"RegionName": "us-east-1",
"Messages": [],
"ZoneName": "us-east-1a"
},
{
"State": "available",
"RegionName": "us-east-1",
"Messages": [],
"ZoneName": "us-east-1c"
},
{
"State": "available",
"RegionName": "us-east-1",
"Messages": [],
"ZoneName": "us-east-1d"
},
{
"State": "available",
"RegionName": "us-east-1",
"Messages": [],
"ZoneName": "us-east-1e"
}
]
}
The table format displays a text/ASCII table of results. This can be useful for generating printable reports:
The text output format only displays the resulting key/value response. No additional formatting or display characters are added.
The CLI tool supports transforming the response from the API with the --query option. This option takes a JMESPath query as a parameter and returns the query result.
Note
JMESPath is a query language for JSON. For more information, visit http://jmespath.org/.
As the query is processed as part of the command, it takes place on the server, not the client. By offloading work to the server, you can reduce the size of the resulting payload and improve response times.
JMESPath can be used to transform the response that you receive:
$ aws ec2 describe-availability-zones \ --output json \ --query 'AvailabilityZones[].ZoneName' [ "us-east-1a", "us-east-1c", "us-east-1d", "us-east-1e" ]
It can also be used to filter the data that is received:
$ aws ec2 describe-availability-zones --output json --query "AvailabilityZones[?ZoneName == 'us-east-1a'].State" [ "available" ]
When performing complex tasks with the CLI tool, it may be easier to pass a JSON object of options. This kind of interaction may signify that you should use one of the AWS software development kits (SDKs).
To generate a sample JSON object that will be accepted, run the command with the --generate-cli-skeleton option:
$ aws ec2 describe-availability-zones --generate-cli-skeleton { "DryRun": true, "ZoneNames": [ "" ], "Filters": [ { "Name": "", "Values": [ "" ] } ] }
You can then copy, edit, and use this object to define your command options without passing lots of individual options. It works best for commands with arrays of options or a variable number of options.
You can also get a preview of the output of a command by calling the command with the --generate-cli-skeleton output option. This can speed up the process of combining CLI commands as you can see a response without actually calling the API:
$ aws ec2 describe-availability-zones --generate-cli-skeleton output { "AvailabilityZones": [ { "ZoneName": "ZoneName", "State": "State", "RegionName": "RegionName", "Messages": [ { "Message": "Message" } ] } ] }
Results returned by the CLI tool are limited to 1,000 resources by default.
This is not normally an issue, but at a certain scale, you may run into pagination issues. A common example is files in an S3 bucket.
Note
If you are absolutely sure you should be seeing a particular resource in a response but cannot, check your pagination. The resource may be included in the matching resources, just not in the part of the response that was returned to you.
The following options allow you to control the number and starting point of the results returned to you from the API:
--page-size: This limits how many resources will be displayed to you, but does not actually limit the number returned. The default number of items (that is, 1,000) will still be processed and returned to you.--max-items: This sets an upper limit on how many items will actually be returned in the response. You may receive fewer items, but you will not receive more than this number.--starting-token: This changes where the response starts. Use this to display subsequent results, beyond the first page.
aws s3 ls --bucket bucket-name --max-items 100 --starting-token None___100You can enable tab-completion of commands, subcommands, and options by configuring the completer included with the CLI tool.
On OS X, Linux, and Windows systems with a bash shell, you can load the completer with the following command:
complete -C 'which aws_completer'awsBy default, the aws_completer program is installed in /usr/local/bin. If your tool is installed to a non-standard location, you will need to find it and change the which aws_completer command to the relevant path.
The following program work nicely with the AWS CLI tool, and may come in handy.
jq is a lightweight tool for processing and transforming JSON. It follows the Unix philosophy of doing one thing and doing it well. It can be found at https://stedolan.github.io/jq/.
Note
While jq and JMESPath are similar, jq is a lot easier to get started with. It also supports transforming JSON into plaintext; JMESPath queries will always return more JSON.
You can pipe JSON results from the CLI tool to it, and easily transform the results for use elsewhere. This example uses jq's property name selectors to convert JSON output to text:
$ aws ec2 describe-availability-zones --output json | jq '.AvailabilityZones[].ZoneName' "us-east-1a" "us-east-1c" "us-east-1d" "us-east-1e"
