Chef Infrastructure Automation Cookbook

You'll need Git installed on your local workstation. Either use your operating system's package manager (such as Apt on Ubuntu or Homebrew on OS X), or simply download the installer from www.git-scm.org.

Git is a distributed version control system. This means that you don't necessarily need a central host to store your repositories. However, in practice, using GitHub as your central repository has proven to be very helpful. In this book, I'll assume that you're using GitHub. Therefore, you need to go to www.github.com and create an (free) account to follow the instructions given in this book. Make sure that you upload your Secure Shell (SSH) key by following the instructions at https://help.github.com/articles/generating-ssh-keys, so that you're able to use the SSH protocol to interact with your GitHub account.

As soon as you have created your GitHub account, you should create your repository by visiting https://github.com while you're still logged in and using chef-repo as the repository name.

Make sure you have wget installed on your local workstation, in order to be able to download the required files from public servers.

Before you can write any cookbooks, you need to set up your initial Git repository on your development box. Chef Software, Inc. provides an empty Chef repository to get you started. Let's see how you can set up your own Chef repository with Git, using Chef's skeleton.

You have downloaded a tarball containing Chef's skeleton repository. Then, you initialized chef-repo and connected it to your own repository on GitHub.

After that, you added all the files from the tarball to your repository and committed them. This makes Git track your files and the changes you make later.

Finally, you pushed your repository to GitHub, so that your co-workers can use your code too.

Let's assume you're working on the same chef-repo repository, together with your co-workers. They cloned your repository, added a new cookbook called other_cookbook, committed their changes locally, and pushed their changes back to GitHub. Now, it's time for you to get the new cookbook downloaded on to your own laptop.

Pull your co-workers' changes from GitHub. This will merge their changes into your local copy of the repository. Use the pull subcommand:

mma@laptop:~/chef-repo $ git pull --rebase

From github.com:mmarschall/chef-repo
 * branch            master     -> FETCH_HEAD
...TRUNCATED OUTPUT...
create mode 100644 cookbooks/other_cookbook/recipes/default.rb

In case of any conflicting changes, Git will help you merge and resolve them.

Let's see how to install the Chef DK on your local workstation using Chef's omnibus installer:

The omnibus installer will download Ruby and all required Ruby gems into /opt/chefdk.

Before being able to use hosted Chef, you need to sign up for the service. There is a free account for up to five nodes.

Visit http://manage.chef.io/signup and register for a free trial or a free account.

I registered as the user webops with an organization short name of awo.

After registering your account, it is time now to prepare your organization to be used with your chef-repo repository.

Carry out the following steps in order to interact with the hosted Chef:

Hosted Chef uses two private keys (called validators):

You need to tell knife where it can find these two keys in your knife.rb file.

The following two lines of code in your knife.rb file tell the knife about which organization to use and where to find its private key. The validation_key is used to allow new clients to authenticate the Chef server before getting their own Client key:

validation_client_name   "awo-validator"
validation_key           "#{current_dir}/awo-validator.pem"

The following line of code in your knife.rb file tells the knife where to find your users' private key. It is used by your local workstation to authenticate the Chef server:

client_key               "#{current_dir}/webops.pem"

Also, the following line of code in your knife.rb file tells knife that you are using hosted Chef. You will find your organization name as the last part of the URL:

chef_server_url          "https://api.chef.io/organizations/awo"

Using the knife.rb file and your two validators knife, you can now connect to your organization hosted by Chef Software, Inc.

You do not need your own self-hosted Chef server, nor do you need to use Chef client local mode in this setup.

This setup is good for you if you do not want to worry about running, scaling, and updating your own Chef server and if you're happy with saving all your configuration data in the Cloud (under the control of Chef Software, Inc.).

Download and install VirtualBox at https://www.virtualbox.org/wiki/Downloads.

Download and install Vagrant at https://www.vagrantup.com/downloads.html.

Install the Omnibus Vagrant plugin to enable Vagrant to install the Chef client on your VM by running the following command:

mma@laptop:~/chef-repo $ vagrant plugin install vagrant-omnibus
Installing the 'vagrant-omnibus' plugin. This can take a few minutes...
Installed the plugin 'vagrant-omnibus (1.4.1)'!

Let's create and boot a virtual node by using Vagrant:

The Vagrantfile is written in a Ruby Domain Specific Language (DSL) to configure the Vagrant virtual machines. We want to boot a simple Ubuntu VM. Let's go through the Vagrantfile step by step.

First, we create a config object. Vagrant will use this config object to configure the VM:

Vagrant.configure("2") do |config|
end

Inside the config block, we tell Vagrant which VM image to use, in order to boot the node:

config.vm.box = "opscode-ubuntu-14.10"
config.vm.box_url = "https://opscode-vm-bento.s3.amazonaws.com/vagrant/virtualbox/opscode_ubuntu-14.10_chef-provisionerless.box"

We want to boot our VM using a so-called Bento Box, provided by Chef. We use Ubuntu Version 14.10 here.

As we want our VM to have the Chef client installed, we tell the omnibus vagrant plugin to use the latest version of Chef client:

  config.omnibus.chef_version = :latest

After selecting the VM image to boot, we configure how to provision the box by using Chef. The Chef configuration happens in a nested Ruby block:

  config.vm.provision :chef_client do |chef|
  ...
  end

Inside this Chef block, we need to instruct Vagrant on how to hook up our virtual node to the Chef server. First, we need to tell Vagrant where to store all the Chef stuff on your node:

    chef.provisioning_path = "/etc/chef"

Vagrant needs to know the API endpoint of your Chef server. If you use hosted Chef, it is https://api.chef.io/organizations/<YOUR_ORG>. You need to replace <YOUR_ORG> with the name of the organization that you created in your account on hosted Chef. If you are using your own Chef server, change the URL accordingly:

    chef.chef_server_url = "https://api.chef.io/organizations/<YOUR_ORG>"

While creating your organization on hosted Chef, you must download your private key. Tell Vagrant where to find this file:

    chef.validation_key_path = ".chef/<YOUR_ORG>—validator.pem"

Also, you need to tell Vagrant as to which client it should validate itself against the Chef server:

    chef.validation_client_name = "<YOUR_ORG>-validator"

Finally, you should tell Vagrant how to name your node:

    chef.node_name = "server"

After configuring your Vagrantfile, all you need to do is run the basic Vagrant commands such as vagrant up, vagrant provision, and vagrant ssh. To stop your VM, just run the vagrant halt command.

If you want to start from scratch again, you will have to destroy your VM and delete both the client and the node from your Chef server by running the following command:

mma@laptop:~/chef-repo $ vagrant destroy
mma@laptop:~/chef-repo $ knife node delete server -y && knife client delete server -y

Alternatively, you may use the Vagrant Butcher plugin found at https://github.com/cassianoleal/vagrant-butcher.

Make sure you have Chef DK installed and a node available for testing. Check out the installation instructions at http://learn.chef.io if you need help here.

Edit your knife.rb file (usually found in the hidden .chef directory) and add the following three lines to it, filling in your own values:

cookbook_copyright "your company"
cookbook_license "apachev2"
cookbook_email "your email address"

Chef will use the preceding values as default whenever you create a new cookbook.

We assume that you have a node called server registered with your Chef server, as described in the Managing virtual machines with Vagrant section in this chapter.

Carry out the following steps to create and use cookbooks:

The chef executable helps you to manage your local Chef development environment. We used it here to generate the cookbook.

Knife is the command-line interface for the Chef server. It uses the RESTful API exposed by the Chef server to do its work and helps you to interact with the Chef server.

The knife command supports a host of commands structured as follows:

knife <subject> <command>

The <subject> used in this section is either cookbook or node. The commands we use are upload for the cookbook, and run_list add for the node.

Make sure that you have the iptables cookbook installed locally and uploaded on your Chef server.

Let's find out how knife can help you to look into a cookbook stored in your Chef server:

The knife cookbook show subcommand helps you understand what exactly is stored on the Chef server. It lets you drill down into specific sections of your cookbooks and see the exact content of the files stored in your Chef server.

Since Chef 11, you can pass patterns to the knife show command to tell it what exactly you want to see. Showing the contents of the iptables_rule definition can be done as follows, in addition to the way we described previously:

mma@laptop:~/work/chef_helpster $ knife show cookbooks/iptables/definitions/*

cookbooks/iptables/definitions/iptables_rule.rb:
#
# Cookbook Name:: iptables
# Definition:: iptables_rule
#
#
define :iptables_rule, :enable => true, :source => nil, :variables => {} do
...TRUNCATED OUTPUT...
end

Make sure you have a cookbook named my_cookbook, and the run_list command of your node includes my_cookbook, as described in the Creating and using cookbooks recipe in this chapter.

Edit the metadata of your cookbook in the file cookbooks/my_cookbook/metadata.rb to add a dependency to the build-essential cookbook:

mma@laptop:~/chef-repo $ subl cookbooks/my_cookbook/metadata.rb
...
depends 'build-essential'
depends 'apache2', '>= 1.0.4'

If you want to use a feature of another cookbook inside your cookbook, you will need to include the other cookbook in your recipe using the include_recipe directive:

include_recipe 'build-essential'

To tell the Chef server that your cookbook requires the build-essential cookbook, you need to declare that dependency in the metadata.rb file. If you uploaded all the dependencies to your Chef server either using berks install and berks upload or knife cookbook upload ..., the Chef server will then send all the required cookbooks to the node.

The second depends call tells the Chef server that your cookbook depends on a version of the apache2 cookbook, which is greater or equal to the version 1.0.4. You may use any of these version constraints with your depends calls:

If you include another recipe inside your recipe, without declaring the cookbook dependency in your metadata.rb file, Foodcritic will warn you:

mma@laptop:~/chef-repo $ foodcritic cookbooks/my_cookbook

FC007: Ensure recipe dependencies are reflected in cookbook metadata: cookbooks/my_cookbook/recipes/default.rb:9

Additionally, you can declare conflicting cookbooks through the conflicts call:

conflicts "nginx"

Of course, you can use version constraints exactly the way you did with depends.

Make sure you have a cookbook named my_cookbook and the run_list of your node includes my_cookbook, as described in the Creating and using cookbooks recipe.

Berkshelf helps you to keep those utility cookbooks out of your Chef repository. This makes it much easier to maintain the cookbooks, which really matter.

Let's see how to write a cookbook by running a bunch of utility recipes and manage the required cookbooks with Berkshelf:

Berkshelf comes with the Chef DK.

We create our cookbook and tell it to use a few basic cookbooks.

Instead of making us manually install all the cookbooks using knife cookbook site install, chef generate creates a Berksfile, besides the metadata.rb file.

The Berksfile is pretty simple. It tells Berkshelf to use the Chef supermarket as the default source for all cookbooks:

source "https://supermarket.chef.io"

It tells Berkshelf to read the metadata.rb file to find all the required cookbooks. This is the simplest way when working inside a single cookbook. Please see the following There's more… section to find an example of a more advanced usage of the Berksfile.

After telling Berkshelf where to find all the required cookbook names, we use it to install all those cookbooks:

berks install

Berkshelf stores cookbooks in ~/.berkshelf/cookbooks, by default. This keeps your Chef repository clutter-free. Instead of having to manage all the required cookbooks inside your own Chef repository, Berkshelf takes care of them. You simply need to check in Berksfile with your recipe, and everyone using your recipe can download all the required cookbooks by using Berkshelf.

To make sure that there's no mix-up with different cookbook versions when sharing your cookbook, Berkshelf creates a file called Berksfile.lock alongside Berksfile. Here, you'll find the exact versions of all the cookbooks that Berkshelf installed:

DEPENDENCIES
  my_cookbook
    path: .
    metadata: true

GRAPH
  apt (2.6.0)
  chef-client (3.9.0)
    cron (>= 1.2.0)
    logrotate (>= 1.2.0)
    windows (~> 1.11)
  chef_handler (1.1.6)
  cron (1.6.1)
  logrotate (1.7.0)
  my_cookbook (0.1.0)
    apt (>= 0.0.0)
    chef-client (>= 0.0.0)
    ntp (>= 0.0.0)
  ntp (1.6.8)
  windows (1.34.8)
    chef_handler (>= 0.0.0)

Berkshelf will only use the exact versions specified in the Berksfile.lock file, if it finds this file.

Finally, we use Berkshelf to upload all the required cookbooks on the Chef server:

berks upload

Berkshelf integrates tightly with Vagrant via the vagrant-berkshelf plugin. You can set up Berkshelf and Vagrant in such a way that Berkshelf installs and uploads all the required cookbooks on your Chef server whenever you execute vagrant up or vagrant provision. You'll save all the work of running berks install and berks upload manually before creating your node with Vagrant.

Let's see how you can integrate Berkshelf and Vagrant.

First, you need to install the Berkshelf plugin for Vagrant:

mma@mma-mbp:~/work/chef-repo (master)$ vagrant plugin install vagrant-berkshelf

Installing the 'vagrant-berkshelf' plugin. This can take a few minutes...
Installed the plugin 'vagrant-berkshelf (4.0.1)'!

Then, you need to tell Vagrant that you want to use the plugin. You do this by enabling the plugin in Vagrantfile:

mma@mma-mbp:~/work/chef-repo (master)$ subl Vagrantfile

config.berkshelf.enabled = true

Then, you need a Berksfile in the root directory of your Chef repository to tell Berkshelf which cookbooks to install on each Vagrant run:

source 'https://supermarket.chef.io'

cookbook 'my_cookbook', path: 'cookbooks/my_cookbook'

Eventually, you can start your VM using Vagrant. Berkshelf will first download and then install all the required cookbooks in the Berkshelf, and upload them on the Chef server. Only after all the cookbooks are made available on the Chef server by Berkshelf, will Vagrant go on:

mma@mma-mbp:~/work/chef-repo $ vagrant up

Bringing machine 'server' up with 'virtualbox' provider...

==> default: Loading Berkshelf datafile...
==> default: Updating Vagrant's Berkshelf...
==> default: Resolving cookbook dependencies...
==> default: Fetching 'my_cookbook' from source at cookbooks/my_cookbook
==> default: Fetching cookbook index from https://supermarket.getchef.com...
...TRUNCATED OUTPUT...

This way, using Berkshelf together with Vagrant, saves a lot of manual steps and gets faster cycle times for your cookbook development.

You'll need to make sure that your local Git repository is clean and does not have any uncommitted changes:

mma@laptop:~/chef-repo $ git status

# On branch master
nothing to commit (working directory clean)

Carry out the following given steps:

The knife command successfully downloaded and imported the mysql cookbook as well as its dependencies.

Knife executes a set of commands to download the desired cookbook and to integrate it with your local repository.

Let's take a look at the output of the knife cookbook site install command again and go through it step-by-step:

The knife cookbook site install command executes all the previously mentioned steps for all the cookbooks the desired cookbook depends on, by default.

Eventually, you end up with a separate branch, the so-called vendor branch, for every downloaded cookbook integrated into your master branch and nicely tagged as shown:

mma@laptop:~/chef-repo$ git branch -a

  chef-vendor-iptables
  chef-vendor-mysql
* master
  remotes/origin/master

This approach enables you to change whatever you like in your master branch and still pull down newer versions of the community cookbook. Git will automatically merge both versions or ask you to remove conflicts manually; these are standard Git procedures.

If you want to integrate the desired cookbook into another branch, use the --branch BRANCH_NAME parameter.

mma@laptop:~/chef-repo [experimental] $ knife cookbook site install mysql –-branch experimental

Installing mysql to /Users/mma/work/chef-repo/cookbooks
Checking out the experimental branch.
Pristine copy branch (chef-vendor-mysql) exists, switching to it.
Downloading mysql from the cookbooks site at version 5.6.1 to /Users/mma/work/chef-repo/cookbooks/mysql.tar.gz
Cookbook saved: /Users/mma/work/chef-repo/cookbooks/mysql.tar.gz
Removing pre-existing version.
Uncompressing mysql version 5.6.1.
removing downloaded tarball
No changes made to mysql
Checking out the experimental branch.
…TRUNCATED OUTPUT…

As you can see, instead of checking out the master branch, the knife cookbook site install command uses the experimental branch now.

You can use the -D switch when running the command in order to avoid downloading all the cookbooks that your desired cookbook depends on.

mma@laptop:~/chef-repo $ knife cookbook site install mysql -D

Installing mysql to /Users/mma/work/chef-repo/cookbooks
Checking out the master branch.
Pristine copy branch (chef-vendor-mysql) exists, switching to it.
Downloading mysql from the cookbooks site at version 5.6.1 to /Users/mma/work/chef-repo/cookbooks/mysql.tar.gz
Cookbook saved: /Users/mma/work/chef-repo/cookbooks/mysql.tar.gz
Removing pre-existing version.
Uncompressing mysql version 5.6.1.
removing downloaded tarball
No changes made to mysql
Checking out the master branch.

You can see that the command stopped after dealing with the mysql cookbook. It has not yet gotten the other cookbooks.

Make sure you have an account at Amazon Web Services (AWS) if you want to follow along and try out the knife-ec2 plugin. There are knife plugins available for most Cloud providers. Go through the There's more... section of this recipe for the list.

Let's see which knife plugins are available, and try to use one to manage Amazon EC2 instances:

Knife looks for plugins at various places.

First, it looks into the .chef directory, which is located inside your current Chef repository, to find the plugins specific to this repository:

./.chef/plugins/knife/

Then, it looks into the .chef directory, which is located in your home directory, to find the plugins that you want to use in all your Chef repositories:

~/.chef/plugins/knife/

Finally, it looks for installed gems. Knife will load all the code from any chef/knife/ directory found in your installed Ruby gems. This is the most common way of using plugins developed by Chef or the Chef community.

There are hundreds of knife plugins, including plugins for most of the major Cloud providers, as well as the major virtualization technologies, such as VMware, vSphere, and Openstack, amongst others.

Make sure you have at least one node registered on your Chef server that is safe to remove.

Let's delete the node and client object to completely remove a node from the Chef server.

To keep your Chef server clean, it's important to not only manage your node objects but to also take care of your client objects, as well.

Knife connects to the Chef server and deletes the node object with a given name, using the Chef server RESTful API.

The same happens while deleting the client object on the Chef server.

After deleting both objects, your node is totally removed from the Chef server. Now, you can reuse the same node name with a new box or virtual machine.

It is a bit tedious and error prone when you have to issue two commands. To simplify things, you can use a knife plugin called playground.

Let's run my_cookbook on your local workstation using Chef client's local mode:

The -z parameter switches the Chef client into local mode. Local mode uses chef-zero—a simple, in-memory version of the Chef server provided by Chef DK—when converging the local workstation.

By providing the -o parameter, you override the run list of your local node so that the Chef client executes the default recipe from my_cookbook.

Chef-zero saves all modifications made by your recipes to the local file system. It creates a JSON file containing all node attributes for your local workstation in the nodes directory. This way, the next time you run the Chef client in local mode, it will be aware of any changes your recipes made to the node.

mma@laptop:~/chef-repo $ knife node run_list add -z laptop 'recipe[my_cookbook]'

mma@laptop:~/chef-repo $ knife upload /

laptop:
  run_list: recipe[my_cookbook]

For the following examples, I assume that you have a node named server and that you have at least one cookbook (I'll use the ntp cookbook) registered with your Chef server.

Let's create a role and see what we can do with it.

You define a role in a Ruby file inside the roles folder of your Chef repository. A role consists of a name attribute and a description attribute. Additionally, a role usually contains a role-specific run list and role-specific attribute settings.

Every node, which has a role in its run list, will have the role's run list expanded into its own. This means that all the recipes (and roles), which are in the role's run list, will be executed on your nodes.

You need to upload your role on your Chef server by using the knife role from file command.

Only then should you add the role to your node's run list.

Running the Chef client on a node having your role in its run list will execute all the recipes listed in the role.

The attributes you define in your role will be merged with attributes from environments and cookbooks, according to the precedence rules described at https://docs.chef.io/roles.html#attribute-precedence.

For the following examples, I assume that you have a node named server in the _default environment and that you have at least one cookbook (I'll use the ntp cookbook) registered with your Chef server.

Let's see how to manipulate environments using knife.

A common use of environments is to promote cookbook updates from development to staging and then into production. Additionally, they enable you to use different cookbook versions on separate sets of nodes and environment-specific attributes. You might have nodes with lesser memory in your staging environment as in your production environment. By using environment-specific default attributes, you can, for example, configure your MySQL service to consume lesser memory on staging than on production.

Be aware that roles are not environment-specific. You may use environment-specific run lists, though.

The node's environment can be queried using the node.chef_environment method inside your cookbooks.

If you want your environments to be under version control (and you should), a better way to create a new environment is to create a new Ruby file in the environments directory inside your Chef repository:

mma@laptop:~/chef-repo $ cd environments
mma@laptop:~/chef-repo $ subl book.rb
name "book"

You should add, commit, and push your new environment file to GitHub:

mma@laptop:~/chef-repo $ git add environments/book.rb
mma@laptop:~/chef-repo $ git commit -a -m "the book env"
mma@laptop:~/chef-repo $ git push

Now, can create the environment on the Chef server from the newly created file using knife:

mma@laptop:~/chef-repo $ knife environment from file book.rb
Created Environment book

There is a way to migrate all the nodes from one environment to another by using knife exec:

mma@laptop:~/chef-repo $ knife exec -E 'nodes.transform("chef_environment:_default") { |n| n.chef_environment("book")

You can limit your search for nodes in a specific environment:

mma@laptop:~/chef-repo $ knife search node "chef_environment:book"

1 item found

Make sure you have at least one cookbook (I'll use the ntp cookbook) registered with your Chef server.

Let's see what happens if we freeze a cookbook.

By using the --freeze option when uploading a cookbook, you tell the Chef server that it should not accept any changes to the same version of the cookbook anymore. This is important if you're using environments and want to make sure that your production environment cannot be broken by uploading a corrupted cookbook.

By changing the version number of your cookbook, you can upload the new version. Then you can make, for example, your staging environment use that new cookbook version.

To support a more elaborate workflow, you can use the knife-spork knife plugin. It helps multiple developers work on the same Chef server and repository without treading on each other's toes. You can find more information about it at https://github.com/jonlives/knife-spork.

You need to have a node registered with your Chef server. It needs to be able to run chef-client without any errors.

Let's see how to start the Chef client in the daemon mode so that it runs automatically.

The -i parameter will start the Chef client as a daemon. The given number is the seconds between each Chef client run. In the previous example, we specified 1,800 seconds, which results in the Chef client running every 30 minutes.

You can use the same command in a service startup script.

Instead of running the Chef client as a daemon, you can use a Cronjob to run it every so often:

user@server:~$ subl /etc/cron.d/chef_client

PATH=/usr/local/bin:/usr/bin:/bin
# m h dom mon dow user command
*/15 * * * * root chef-client -l warn | grep -v 'retrying [1234]/5 in'

This cronjob will run the Chef client every 15 minutes and swallow the first four retrying warning messages. This is important to avoid Cron sending out e-mails if the Chef server is a little slow and the Chef client needs a few retries.

user@server:~$ sudo killall -USR1 chef-client

Running chef-shell is straightforward.

Chef-shell starts an interactive Ruby Shell (IRB) session, which is enhanced with some Chef-specific features. It offers certain modes, such as attributes_mode or recipe_mode, which enable you to write commands like you would put them into attributes file or recipes.

Entering a resource command into the recipe context will create the given resource, but not run it yet. It's like Chef reading your recipe files and creating the resources but not yet running them. You can run all the resources you created within the recipe context using the run_chef command. This will execute all the resources on your local box and physically change your system. To play around with temporary files, your local box might do, but if you're going to do more invasive stuff, such as installing or removing packages, installing services, and so on, you might want to use chef-shell from within a Vagrant VM.

Not only can you run chef-shell in standalone mode but you can also in Chef client mode. If you run it in Chef client mode, it will load the complete run list of your node and you'll be able to tweak it inside the chef-shell. You start the Chef client mode by using the run it --client parameter:

mma@laptop:~/chef-repo $ chef-shell --client

You can configure which Chef server to connect it to in a file called chef-shell.rb, in the same way as you do in the client.rb file on your local workstation.

You can use chef-shell to manage your Chef server, for example, listing all nodes:

chef > nodes.list
[node[my_server]]

You can put breakpoints into your recipes. If it hits a breakpoint resource, chef-shell will stop the execution of the recipe and you'll be able to inspect the current state of your Chef run:

breakpoint "name" do
  action :break
end