Getting Started with Terraform: Manage production infrastructure as a code, Second Edition

The amount of servers used by almost any project is growing rapidly mostly due to increasing adoption of cloud technologies. As a result, traditional ways of managing IT infrastructure become less and less relevant.

The manual approach fits well for a farm of a dozen, perhaps even a couple of dozen of servers. But when we're talking about hundreds of them, doing anything by hand is definitely not going to play out well.

It's not only about servers, of course. Every cloud provider gives extra services on top, be it a virtual networking service, object storage, or a monitoring solution, which you don't need to maintain yourself. These services function that a Software as a Service (SaaS). And actually, we should treat various SaaS products as part of our infrastructure as well. If you use New Relic for monitoring purposes, then it is your infrastructure too, with the difference that you don't need to manage servers for it yourself. But how you use it and whether you use it correctly is up to you.

No surprises, companies of any size, from small start-ups to huge enterprises, are adopting new techniques and tools to manage and automate their infrastructures. These techniques eventually got a new name: Infrastructure as Code (IaC).

Dated something 2009, the Infrastructure as Code term is all about approaching your IT-infrastructure tasks the same way you develop software. This includes the things similar to the following:

Infrastructure as Code is a foundation for DevOps culture because both operations and developers approach their work in the same way, and by following the principles laid out before, they already have some common ground.

This is not to say that if your infrastructure is treated as code, then the border between development and operations becomes so blurry that the whole existence of this separation can become eventually quite questionable.

Of course, the introduction of Infrastructure as Code requires new kinds of tools.

What is infrastructure code specifically? It depends highly on your particular infrastructure setup.

In the simplest case, it might be just a bunch of shell scripts and component-specific configuration files (Nginx configuration, cron jobs, and so on) stored in source control. Inside these shell scripts, you specify exact steps computer needs to take to achieve the state you need:

This is what we call procedural programming. It's not bad. For example, build steps of Continuous Integration tools such as Jenkins that are a perfect fit for a procedural approach-€”after all, the sequence of command is exactly what you need in this case.

However, you can only go far with shell scripts when it comes to configuring servers and higher-level pieces. The more common and mature approach these days is to use tools that provide a declarative, rather than a procedural, way to define your infrastructure. With declarative definitions, you don't need to think how to do something; you only write what should be there.

Perhaps the main benefit of it is that rerunning a declarative definition will never do the same job twice, whereas executing the same shell script will most likely break something on the second run. The proper configuration management tool will ensure that the server is in the exactly same state as defined in your code. This property of modern configuration and provisioning tools is named idempotency.

Let's look at an example. Let's say that you have a box in your network that hosts a packages repository. For some reason, instead of using DNS server, you want to hardcode the IP address of this box to the /etc/hosts file with the domain name repository.internal.

Not a complex task to do, given that you only need to add a new line to the /etc/hosts file. To achieve this, you could have a script like the following:

echo 192.168.0.5 repository.internal >> /etc/hosts/hosts

Running it once will do the job: required entry will be added to the end of the /etc/hosts file. But what will happen if you execute it again? You guessed right: exactly the same line will be appended again. And, even worse, what if the IP address of the repository box will changes? Then, if you execute your script, you will end up with two different host entries for the same domain name.

You can ensure idempotency yourself inside the script with the high usage of conditional checks. But why reinvent the wheel when there is already a tool to do exactly this job? It would be so much better to just define the end result without composing a sequence of commands to achieve this.

And that is exactly what configuration management tools such as Puppet and Chef do by providing you with a special Domain Specific Language (DSL) to define the desired state of the machine. The certain downside is the necessity to learn a new DSL: a special small language focused on solving one particular task. It's not a complete programming language, neither does it need to be; in this case, its only job is to describe the state of your server.

Let's look at how the same task could be done with the help of a Puppet manifest:

host { 'repository.internal': 
  ip => '192.168.0.5', 
} 

Applying this manifest multiple times will never add extra entries, and changing the IP address in the manifest will be reflected correctly in host files, changing the existing entry and not creating a new one.

Now you should better understand that, when it comes to configuration management, tools that provide the declarative way of doing things are preferred.

Modern configuration management tools such as Chef or Puppet completely solve the problem of setting up a single machine. There is an increasing number of high-quality libraries (be it cookbooks or modules) for configuring all kinds of software in an (almost) OS-agnostic way. But configuring what goes inside a single server is only part of the picture. The other part, which is located a layer above, also requires new tooling.

Quite often, servers are only one part of infrastructure. With cloud platforms such as Amazon Web Services (AWS), Google Cloud Platform, and OpenStack advancing more and more, there is an increased need for automating and streamlining the way people work with the services these platforms provide. If you rely heavily on at least one cloud provider for major parts of your project, you will start meeting challenges in applying consistent patterns of their usage.

The approach of modern configuration management tools, while having been around for quite some time and having been adopted by many companies, has some inconveniences when it comes to managing anything but servers.

There is a strong likelihood you would want these patterns to be written once and then applied automatically. Even more, you need to be able to reproduce every action and test the result of it, following the aforementioned Infrastructure as Code principles. Otherwise, working with cloud providers will either end up in so-called ClickOps, where you work with infrastructure primarily by clicking buttons in the web interface of a cloud provider, or you will script all the processes by using APIs of this provider directly. And, even if scripting APIs sounds like a big step towards true Infrastructure as Code, you can achieve much more using existing tools for this exact task.

There is a certain need for a configuration tool that operates one level higher than a setup of a single server; a tool that would allow writing a blueprint that would define all of the high-level pieces at once: servers, cloud services, and even external SaaS products. A tool like this is called given a different name: infrastructure orchestrator, infrastructure provisioner, infrastructure templating, and so on. No matter what you call it, at some point in time, your infrastructure will really need it.

Before proceeding to the existing solutions, let's lay out a list of the most important requirements for a tool such as this, so we are able to choose one wisely.

Now that we have a problem to solve and a list of requirements the tool that should solve the problem, we can go into the specifics of the different existing tools.

Terraform is an open source utility, created by the HashiCorp company, the same company that created Vagrant, Packer, Consul, and other popular infrastructure tools. It was initially released in July 2014, and since then, has come a long way to become one of the most important tools for infrastructure provisioning and management.

This is how Terraform is described by HashiCorp:

Terraform easily fits most of the requirements listed here:

Over the next seven chapters, we will learn how to use Terraform and all of its features.

This is a book about Terraform, and you will learn everything that there is to learn about this tool. There are two main parts to this book, split into six chapters of pure learning.

In the next three chapters, we will learn the basics. In Chapter 2, Deploying First Server, the next one, you will learn the basics of Terraform, the main entities it uses, and how to deploy our first server with it. We will also get a short introduction to AWS EC2.

In Chapter 3, Resource Dependencies and Modules, we will discover how exactly Terraform operates with its resources and how to refactor our code. In Chapter 4, Storing and Supplying Configuration, you will learn all the possible ways you can configure your templates with the various APIs Terraform provides.

If you are already familiar with the Terraform basics, Chapter 2, Deploying First Server, to Chapter 4, Storing and Supplying Configuration, might be a bit boring for you. They are about how to use this tool as a first-time user, and they don't cover many advanced topics that you will get to once you run Terraform in production. Feel free to skip the next three chapters if you, already used Terraform. For advanced topics, head over to Chapter 5, Connecting with Other Tools, Chapter 6, Scaling and Updating Infrastructure, and Chapter 7, Collaborative Infrastructure.

In Chapter 5, Connecting with Other Tools, you will learn how to connect Terraform with many different tools, from configuration management to infrastructure testing tools. We will find out how to provision and reprovision machines and how to use Terraform alongside literally any other tool.

In Chapter 6, Scaling and Updating Infrastructure, we will cover infrastructure updates with Terraform, from the very simple cases (such as changing one property of a non-essential resource) to complex upgrade scenarios of whole clusters of machines.

Finally, in Chapter 7, Collaborative Infrastructure, you will learn how to collaborate on infrastructure work with Terraform. We will also master integration testing for Terraform environments.

Be prepared: this book is not only about Terraform. It's about Infrastructure as Code and various topics surrounding it, such as Immutable Infrastructure. Terraform will be the main tool we study, but definitely not the only one. Configuration management tools, testing tools, half a dozen small helper utilities, and the same amount of AWS services; get ready to learn the whole toolset required to embrace Infrastructure as Code because, as you will soon notice, Terraform is a tool that must be supported by other software.

In the final chapter, Chapter 8, Future of Terraform, we will run through multiple topics related to Terraform which did not make it into the other chapters. That chapter, also includes a non-conventional piece on the future of Terraform, which you may or may not want to read before proceeding to learn it.

So, without further delay, let's proceed to creating our first server with Terraform.

In this chapter, you learned a lot about Infrastructure as Code principles and some tools that allow you to leverage them. There are many existing mature tools that take care of configuring what goes inside a single server, but there are not that many options when it comes to defining one level above a single server. We also listed the requirements for a tool that would take care of configuring this higher level. Then, we came to the conclusion that Terraform meets many, if not all, of these requirements. In the next chapter, we will finally get our hands dirty, install Terraform, and get to know how to use it to create a single AWS EC2 server.