Chapter 7. Collaborative Infrastructure
By this chapter, you've learned how to create and manage your infrastructure with Terraform. However, all the topics we have discussed apply only to a single-person operations department. If you are the only one using Terraform in your team, then you have all the knowledge already. Eventually, operations teams will reduce in size, and what required a dozen system admins in the past will require only a couple of them, the ones that are experienced in both operations and software development. Even then, it's not a single person, but at least a couple : having just one infrastructure engineer in a company is an example of a single point of failure.
And when you have multiple colleagues working on Terraform templates, you have a whole new package of problems to solve. How do you store your templates? How do you organize and split them? Where do you store them? And where do you store the state file? How do you roll out changes to production? And how do you...
Version control with Git 101
Note
Feel free to skip this part if you are already familiar with version control and Git specifically.
Version Control System (VCS) simplifies work with constantly changing information, such as code. It allows us to store multiple versions of the same file, easily switch between them, and check who is responsible for which change. The most popular VCS today is Git, initially created to support Linux kernel development.
A VCS such as Git has many benefits:
- You have access to all versions of all files in the Git repository at any time; it's almost impossible to lose any part of a piece of code or a previous state of the code.
- Multiple developers can work on one project at the same time without interfering with each other's code and without fear of losing any changes made by colleagues. In Git, the possibilities of collaborative work are unlimited.
To create a repository, you've got to run git init in the project folder. To add files in it, first use git add file_name...
Traditionally, code in technical books uses GitHub for a good reason: everyone knows it, and it's free for open source (or just public) repositories. We are going to use GitLab though. First, it's free for both public and private projects. Second, it has some features that GitHub lacks, and we will need them for this chapter: more on this later.
Note
You could skip this section as well, but better if you don't. We will go through all the files that we have created in previous chapters and remove everything not needed.
This means that before proceeding further, you will need to get yourself an account at https://about.gitlab.com/ (you can use your GitHub account to log in to GitLab with just few clicks).
We will start by doing a revision (see what I did there?) of all the files we have so far. All the code written previously in the book will be publicly available on GitLab.
Go to your directory...
Protecting secrets in a Git repository
Terraform doesn't provide any built-in way of securing your state file. Neither is there a way to secure only some part of it or even provide encrypted data inside your templates. And it's a shame because, sooner or later, you will have to use some kind of secrets with your templates: passwords, API keys, and others. If you plan to store your state file in the git repository, it's important to protect it. The easiest solution is to encrypt the whole state file, store the encrypted version in the repository, and distribute the key for decryption with your team members.
You could make this task easier with the help of a tool named terrahelp. Terrahelp is a small CLI written in Go that simplifies the encryption and decryption of your Terraform state files (and not only the state files). It has a nice integration with Vault, yet another HashiCorp tool, this time in order to manage secrets. Don't worry, we won't use Vault, it's rather a complex tool that...
Storing state files remotely
As you know, by default, Terraform will store the state file on your local disk and you have to figure out yourself how to distribute it within your team. One option you learned is to store it in the git repository: you get the workflow, you get the versioning and you even get some level of security on top. But there is also a concept of remote state provided by Terraform.
The idea is that, before you start applying your templates, you configure a remote storage. After that, your state file will be pulled and pushed from a remote facility. There are 11 backends for your state provided by Terraform: Consul, S3, etcd, Atlas, and others. You will learn how to use Simple Storage Service (S3) for this purpose.
Note
Atlas is a commercial offering from HashiCorp. One part of it is named Terraform Enterprise: it combines secure remote state storage, versioning of state file changes, logs of Terraform runs, and some other features. It is well integrated with GitHub. You...
Connecting remote states together
Up until now, we naively stored all of our Terraform code in a single repository. We had a single template responsible for creating a network, routes, virtual machines, security groups, and everything else. It works pretty well, provided you have a single application with modest infrastructure around it. A single VPC, a few subnets, a small database, and a couple of instances: with this scale, there are few reasons to go beyond the single repository for all the infrastructure templates.
If you are part of a large organization, this approach can get you only so far. Companies that heavily rely on AWS tend to have dozens of use cases for many, various services. Only the IAM service has quite a few entities to manage: roles, policies, users, groups, and so on. Normally, there are many roles for different servers and even more policies for these roles. The network is also kind of complicated; at the very least, you would have one VPC per environment or even one...
We've stored the application module in the very same directory where our main Terraform template resides. It makes it impossible to reuse: if there is a new application in the company that requires the same infrastructure (meaning the same module), then we cannot easily use it.
Remember the source attribute of the module?
module "mighty_trousers" {
source = "./modules/application" Well, it turns out that it doesn't have to be a path to a local directory. In fact, there are multiple supported sources for modules:
- GitHub
- BitBucket
- Generic Git and Mercurial repositories
- HTTP URLs
- S3 buckets
Storing modules in one of these destinations allows us to have a collection of reusable components. We can even version our modules, just like system packages or programming language libraries.
As we are deep into GitLab already, let's create yet another repository and call it packt-terraform-app-module. As always, all the code written in this chapter is available on GitLab at https:/...
Locking state files with Terragrunt
Let's say you have your application template and a team of five people working on it. One Monday morning you decide to change a minor thing, such as the security group, and at the same time your colleague, sitting in a room next to you, decides to change a disk size for instances. Being confident that you are the only ones running the terraform apply command at this moment, you both do terraform apply, push changed state file to the git repository (or to remote storage like S3), and end up in a total disaster.
If your state file is stored in git, then you will meet the merge conflict: not too bad, you can try to resolve it by hand, and you will still be able to see who changed what. If you use a remote backend for the state file, then things are going south. Which state file is now inside the remote storage? And where do the changes of another Terraform run go?
It is dangerous to work on the same state file in a team, because there is no locking out of the...
Moving infrastructure updates to the CI pipeline
Remember how we started this book with a discussion of infrastructure as code concepts? Well, if we want to go further treating infrastructure as a real code, then we could (and even should) apply all the same best practices currently existing in software development, and Continuous Integration is a big part of it. The idea behind CI (in case you missed all the buzz about it a few years ago) is to be able to test, build, and deploy your code regularly and automatically. The way it works is by using special software that takes care of all the tasks of making your software ready for production. You only need to define which tasks exactly are part of your CI and how to execute them.
Do you remember that we chose GitLab over GitHub due to some features that GitHub lacks? The most important feature that GitLab has completely integrated into all development workflows and that GitHub doesn't have at all, is GitLab CI. Yes, you can use Travis CI or...
Integration testing of Terraform modules
In one of the previous chapters on making various tools play well with Terraform, we already took a quick look at running infrastructure tests. Back then we used Inspec to run a test against the single EC2 instance. A few chapters forward, and now we have much more complex Terraform setup on our hands; one that is split across four repositories.
If we were to consider ourselves old-fashioned traditional system administrators, we would be quite happy with what we have achieved by now. But a good software developer (and if we are doing infrastructure as code, then we are already software developers, regardless of our previous experience) would never leave any code without proper tests. And what we wrote in the past is nothing like a proper test.
But what should we test? We must not run tests against the production environment (the one we just configured GitLab CI for), and it is meaningless to test VPC and IAM repositories in isolation. So the only good...
It's been another long chapter to digest and there has been a ton of new things to learn and try. You started by learning Git and how to organize work though Git branches, remote repositories, and code review. You learned how to easily store secrets in a setup like this with git-crypt. After this, we took a look at the remote storage of state files for Terraform, and at various methods to split the Terraform code inside the organization.
As part of this, we wrote our first completely remote Terraform module, refactored the whole IAM and VPC management away from the main repository, and connected it all nicely in a small, and clean template. To avoid conflicts and to better structure the infrastructure work, we set up the Terragrunt utility and learned how to use it too.
We took the whole infrastructure as code idea to the extreme by introducing a complete Continuous Integration pipeline for the infrastructure (and learned a bit of GitLab CI). As a final battle, we even created a real...
