Packer is a free and open source extensible software tool that takes your desired OS and container configurations and builds them simultaneously for the easy testing and management of complex system and application images and artifacts. If you ever find yourself in an environment where multiple custom system disks or cloud AMIs must be consistently maintained and adjusted to boot VMs or run containers, then Packer is here to simplify your life as you automate configuration through code.

This chapter is a very high-level introduction for those unfamiliar with Packer. It explains how Packer is not a service but a tool that can be manually run or inserted into an automation pipeline. It also describes how Packer can supplement Terraform to dramatically simplify anything from complex hybrid or multi-cloud deployments to on-premises private cloud or even local VMs on a development machine.

In this chapter, we will cover the following topics:

• Packer architecture, which describes how the Packer binary is distributed and developed and how Packer works with templates, builders, and provisioners at a high level
• History of Packer, which is important to understand why Packer was needed in the first place and what business problems it solves
• Who uses Packer?, which lists what types of users Packer has today, including everything from small academic labs to large-scale enterprise organizations and software vendors
• Alternatives to Packer, which is a section that describes industry alternatives and other tools that offer image management and how they compare to Packer at a high level
• Installing Packer, which covers how easy it is to install Packer on most environments, whether servers, cloud instances, or local laptops
• HCL versus JSON, which is a very high-level description of JSON and HashiCorp Configuration Language (HCL) and how Packer supports either standard for templates

For this chapter, you should have a basic understanding of JSON and HCL2 domain specific languages. You won't need to try any sample code for this chapter but if you choose to follow along you may use any device that supports running the Packer binary. This includes any laptop or small device running Linux, macOS, or Windows. As we focus a lot of examples on Linux, it may be useful to run Linux or use a cloud resource running Linux at minimum.

Packer itself is a fairly simple binary written in Go. It supports plugins for various inputs and outputs. The plugins that translate your configuration and scripts into artifact outputs are called builders. Common builders include common hypervisors such as VMware, QEMU, VirtualBox, AWS, GCP, and Microsoft Azure. Builders also include multiple container image formats, including LXC, LXD, Docker, and Podman. Many plugins have been contributed by the community and we will cover how you can write your own in a future chapter.

The bit of code you write to tell Packer what to do is called a template. Early versions of Packer expected your template to be written in JavaScript Object Notation (JSON). As of Packer version 1.7.0, both JSON and HashiCorp Configuration Language version 2 (HCL2) are supported, with the latter being preferred. We will cover both formats and how you can migrate a JSON template into an HCL2 template shortly.

Provisioners are tasks or resources that should be applied to your image before packaging. By default, each builder in your template takes each provisioner. Take an example where you want to build a system image with your application across AWS, Azure, and GCP. All you need to do is define your list of builders for AWS, Azure, and GCP and include a single provisioner that uploads your application.

A build job is what runs the Packer build command with your template. Normally, this forks a parallel process for every builder you specify in your template. A build can happen simultaneously across VMware, AWS, Azure, GCP, or other builders while Packer tracks the results and reports any errors. When all builders finish or end in an error, the job is done and the Packer process terminates. Optionally, Packer may compress output images before terminating, to save space.

The origins of Packer can be found in HashiCorp’s Vagrant product. Vagrant was originally a Ruby project to select from a set of standard OS images, boot one or more VMs, and automatically configure them once booted. Vagrant allowed for rapid environments for development with an extensible framework to support multiple virtualization platforms, such as VirtualBox, VMware, and QEMU.

When managing multiple environments for multiple teams, one needs to strike a balance of build time versus runtime. Provisioning resources is quick and easy when everything comes in a pre-built package or artifact, but purpose-built artifacts for every use case take up quite a bit of storage. What resources will be common across an organization and which might be deployed in different ways when they are consumed? Building multiple gold images for Vagrant or cloud environments becomes a challenge at scale. Packer was built to simplify this and it works very efficiently. It can be run simply on your own computer or it can be inserted into automation jobs and pipelines. We will cover all of these use cases in this book and show you how easily Packer can simplify your image maintenance both locally and in the cloud. A team that needs identical images built across multiple regions, multiple clouds, and possibly even local infrastructure may require complex image management. Each region within each cloud may need multiple versions of an image to be maintained, based on the OS, applications deployed, and custom configuration. Keeping all environments consistent often creates exponential complexity. Imagine each line in this diagram represents a combination that requires an image to be built and maintained:

Figure 1.1 – Managing multiple applications across multiple environments can be complex

Many people will attempt to manage complex environments like this one using purely provisioning tools such as Vagrant and Terraform, which can actually result in more complexity in the end. A minor change to a Terraform provisioner can result in an entire environment being destroyed and rebuilt. It’s important to start with a good image strategy before provisioning to simplify things at runtime. Often, a single Packer template can be used to satisfy all of the preceding combinations.

Packer was also the first HashiCorp project written purely in Go, also known as Golang, the modern programming language created by Google. Go is an optimized compiled language that generates simple statically linked binaries using a community of open source projects. A lot of management tools like Packer tend to be written in a scripting language such as Python or Ruby so that they can be easily ported and customized. Even Vagrant was initially written in Ruby. Scripting languages such as Ruby tend to not perform as well as precompiled Go binaries. Scripting languages are also prone to dependency deprecation and complexity. If you download a Packer binary, everything you need to run is self-contained. You won’t run into an issue where an old OS version of glibc or Python prevents the binary from running. You also won’t have memory leaks or buffer vulnerabilities as Go manages its own memory via garbage collection. Golang has since been the language of choice for HashiCorp projects, including Vagrant, which was rewritten in Golang for consistency. If you don’t know how to write Go, there is no need to worry. You won’t need to write Go to use Packer unless you want to write a plugin or add a feature. We will cover how to do this in Chapter 12, Developing Packer Plugins.

You can also find books on Go from Packt here: https://www.packtpub.com/gb/tech/go?released=Available&language=Go.

Packer is a purely open source tool for HashiCorp but that doesn’t mean that enterprise customers don’t use it. Packer is used to build images in private networks and public clouds around the world, covering many industries from investment banking to universities and students. Individuals and small teams often use Packer to maintain a set of disposable system images for mixed estates, including Mac, Windows, Linux, and serverless cloud applications. Large teams and organizations may use automation or continuous deployment pipelines for Packer to rebuild a set of images when certain events or edits occur. The beauty of Packer is it behaves the same whether it is running in a multi-cloud Fortune 500 firm or running on a laptop in a coffee shop. You can even run Packer on a low-power commodity ARM device such as Raspberry Pi. The difference between a coffee shop laptop and an enterprise deployment really comes down to security and best practices, which we’ll cover in Chapters 6-8.

The open source community has a great variety of sample templates, so you usually don’t need to start one from scratch. Search the Packer documentation page for samples, as well as GitHub. Unlike Vagrant, which has a public registry of source images, Packer requires the user to provide base images.

Terraform users find Packer valuable for any projects that use VM deployments in a hybrid cloud environment. Properly prepared images will dramatically ease VM provisioning with Terraform. More importantly, cloud-native tooling that may provision instances dynamically, such as autoscale groups or failover routines, will not inform Terraform about their activity. Having a proper VM autoscale group deployed with Terraform still requires a standard image for the cloud to scale.

Image management has been a challenge for years. Packer is certainly not the first tool to address these difficulties. Tools such as Solaris JumpStart or Red Hat Kickstart have been used to codify VM installation. These can be used in conjunction with Packer to build uniform images across platforms. Packer may use a kickstart to deploy a Linux platform from standard media but then use provisioners to deploy tooling identically across Linux and Windows environments. Docker Compose and Buildah are also modern tools for building specialized container images. Often, specialized community tools such as this can supplement Packer while letting Packer provide a more general-purpose building tool to bring complex mixed environments into one single template. Red Hat Enterprise Linux users have the option of leveraging Red Hat Satellite for platform standardization using a combination of Kickstart, Cobbler, and Puppet.

Historically, simple scripting has been used for early infrastructure as code strategies. If configuration can be scripted, it can be version controlled and used to build and test images captured with either virtualization tools or image tooling such as Norton Ghost.

Packer is freely available via many options depending on your computer. You can download the full source code at any time via GitHub. In most cases, there is no need to compile your own release. Official binaries are available on HashiCorp’s release page: https://releases.hashicorp.com. The best way to install is via OS releases:

1. If using Brew for Mac, run the following commands to enable the HashiCorp tap and install Packer:

   brew tap hashicorp/tap
   brew install hashicorp/tap/packer

2. For RPM-based Linux distributions, use YUM or DNF to enable the HashiCorp repo for RPM-based Linux using these commands:

   sudo dnf config-manager --add-repo https://rpm.releases.hashicorp.com/fedora/hashicorp.repo
   sudo dnf -y install packer

3. For DEB-based Linux distributions, enable HashiCorp’s APT repo and install Packer using these commands:

   curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo apt-key add -
   sudo apt-add-repository "deb [arch=amd64] https://apt.releases.hashicorp.com $(lsb_release -cs) main"
   sudo apt-get update && sudo apt-get install packer

4. Windows users can use Chocolatey to install Packer using the following command:

   choco install packer

The OS packaging contains secure signed binaries that get verified by packaging. Downloading the binaries from HashiCorp’s releases page manually requires that downloads be verified manually with checksums before use. This verification ensures the build comes from HashiCorp and doesn’t have any compromised code. Never use a Packer release in production without verifying its signature.

It’s good to have some basic background on the three coding formats supported by Packer templates. JSON is a descriptive language that uses blocks to declare a data structure. A JSON document may use an optional schema that is a secondary JSON document that lists the structure for the writer to follow. Since version 1.7, Packer actually supports two versions of JSON, so it’s important to know how to identify them by file extension when coming across older templates. Legacy templates end in just .json, whereas new templates end in .pkr.json, and both options use different schemas or styles. HCL is HashiCorp’s own syntax, which has a few more features than JSON but also a few limitations:

Table 1.1 – Comparison of JSON and HCL code for Packer

The good news is, Packer supports both HCL and JSON and also has a helpful tool to convert an existing JSON template into an HCL template automatically. HCL may support schemas in the future, but currently, its open format features also help make it more flexible and easier to read than JSON in some cases. Let’s start with some examples of both JSON and HCL2 Packer templates. Note there are actually two versions of the JSON schema supported by Packer. The one you use must be reflected in the file extension when you save your template. Legacy JSON templates just end in *.json and are supported for existing templates in the community. Newer JSON templates should be written in HCL2.

Example legacy JSON

The following sample is an excerpt from a legacy JSON template used to build an image on VMware. You may encounter these in older examples and Packer still supports them for backward compatibility. Note that some JSON strings contain Go-style templating, indicated by double braces, {{ }}. Adding comments is not an option in JSON, so it is difficult to document your code. This code starts with a CentOS 7.8 image, boots it on VMware as specified by a builder, and then uses a provisioner to upload a script and another provisioner to run that script.

JSON schemas provide a way to describe the possible options for a desired JSON document, and can help guide a coder with suggestions, auto-completion, and type checking while building a template. Schemas can also generate WYSIWYG editors, which allow automatic menus and designers for those who don’t want to write code manually. Partial community schemas for Packer templates have been written by the author and are available at https://github.com/jboero/hashicorp-schemas/blob/master/JSON/packer/1.5/template.json. These schemas are community-driven, not created by HashiCorp engineers. Note that these template samples won’t build for you unless you specify a compatible base image. We will actually cover a practical example in the next chapter. A sample template in HCL2 is given in the following code block. We will break down this template line by line in the coming chapters. Optional variables can be declared to help make templates reusable. These definitions look like this and let you define whatever variables you like. Here, there are three variables with default values declared that will be used in builder declarations:

variable "base_url" {
  type        = string
  default     = "https://my-source/image.iso"
  description = "URL for our base image"
  sensitive   = false
}

Variables in Packer’s HCL2 format also offer optional validation blocks. This is helpful for limiting what you can assign to the variable. For example, the base_url variable in the preceding example is a URL and we want to restrict it to take only values starting with https, we can specify this using this validation block:

validation {
  condition     = substr(var.base_url, 0, 5) == "https"
  error_message = "URLs must start with https"
}

There are many variables that come built into Packer for each build or source. These give access to dynamic values, such as the unique identifier for the build, name, and ID of the build resource. This is helpful when you want to inject aspects about the build itself into actions or provisioners performed in each environment. For example, if you want to save the Packer build UUID into the image via a file such as /etc/packerbuild, you can reference the build.PackerRunUUID variable. A list of the build and source variables can be found in Packer’s contextual variable documentation: https://developer.hashicorp.com/packer/docs/templates/hcl_templates/contextual-variables.

Builders are plugins used to declare an environment for image building, such as VMware, VirtualBox, QEMU, and Docker. As of Packer version 1.7, templates declare an instance of a builder as a source. In this sample, we declare one builder of the VMWare ISO type with minimal settings to connect our VM. Notice the previous variables are inserted into strings using the {{ }} templating syntax. HCL also supports direct variable usage without strings. A builder says nothing about how your image should be customized. It only tells Packer what kind of environment to run provisioners on to customize your image. Take this example:

source "vsphere-iso" "example" {
  iso_url = var.base_url
  iso_checksum = var.base_checksum
  ssh_username = "packer"
  ssh_password = "packer"
  shutdown_command = "shutdown -P now"
  boot_command = [
    "<esc><wait>",
    "vmlinuz initrd=initrd.img ",
    "<enter>"]
  boot_key_interval = "1ms"
  boot_wait = "1s"
  cpus = 8
  memory = 8192
  disk_size = 4000
}

Provisioners are the magic of Packer. These are customizations, resources, or scripts that should be run on all of the builders to preconfigure everything you expect in the image. Once all of the provisioners are finished, Packer saves the image as configured in the builder. Here, there are two provisioners. The first is a script called install.sh, which we upload into the builder from a local directory, ./http/install.sh. Then, the second provisioner is a shell command to run that script:

provisioner "file" {
  destination = "/tmp/install.sh"
  Source = "./http/install.sh"
  direction = "upload"
}
provisioner "shell" {
  inline = ["sudo bash –x /tmp/install.sh"]
}

Packer can be used to build or simply validate this JSON document as a valid template. Note that JSON templates require a root document. Everything is nested within a single set of braces, also known as a code block. This differs from HCL, which requires no root document or block.

Example PKR.JSON

When Packer added HCL2 support, it restructured how templates are structured. There is an additional JSON option that mirrors this HCL2 format. Builders are instead defined as sources and then a build job lists which sources you would like to include in the build. It may be a little confusing if you are used to legacy JSON support. Packer will select whether your JSON file uses the legacy or new schema by its file extension. For example, template.json uses the legacy schema, as used in the preceding example, whereas template.pkr.json would tell Packer to use the new schema of sources. HCL2 is still the recommended way to build new templates, though JSON support still offers some nice automation options for IDEs and UI wizards, which we’ll discuss in Chapter 2, Creating Your First Template. The equivalent example in pkr.json format is listed in the book’s GitHub repo: https://github.com/PacktPublishing/HashiCorp-Packer-in-Production/blob/main/Chapter01/Sample.pkr.json.

Example HCL

Here, I have taken the previous legacy JSON template and migrated it to HCL2 via Packer’s built-in packer hcl2_upgrade [template.json] command. I have also added some comments to explain what’s happening. HCL supports three comment types: //, /*, and #. I’ve included examples of all of these types in the following snippet, but it’s best to choose one standard and be consistent. HCL has no root object requirement but the structure varies a bit from the JSON version. HCL also supports here docs, also known as here documents, which can help you embed files such as our provisioner script directly into the template. These are often indicated by an <<EOF flag or a similar delimiter. The fully converted template with additional comments added manually is shown here. HCL2 can look quite a bit different than JSON. Variables are declared one at a time like in the following example:

variable "checksum" {
 type    = string
 default = "087a5743dc6fd6...60d75440eb7be14"
}

In addition, each builder is declared separately as a source. Then, a build job lists the sources and provisioners desired:

build {
  sources = ["source.vmware-iso.autogenerated_1"]
  provisioner "file" {
    destination = "/tmp/install.sh"
    direction   = "upload"
    source      = "./http/install.sh"
  }
  provisioner "shell" {
    inline = ["sudo bash -x /tmp/install.sh"]
  }
}

This HCL2 template provides the same details as the JSON version earlier. It has been automatically converted by Packer and commented to provide more detail. In the next chapter, we will break down every line of this template to explain what each value means in detail.

This chapter gave a very high-level overview of the HashiCorp tool called Packer. It’s actually a very simple tool that delivers powerful results when working with image management at any scale. As a tool, Packer is used when needed rather than as a service that listens for tasks. You can build templates that deliver flexible, scalable, standardized images around the world or in your own private data center. We’ve explored a few basic sample templates, but the best way to learn how Packer works is to jump in with building your first template, which we’ll cover in the next chapter.