Chef Infrastructure Automation Cookbook

2 (1 reviews total)
By Matthias Marschall
  • Instant online access to over 7,500+ books and videos
  • Constantly updated with 100+ new titles each month
  • Breadth and depth in over 1,000+ technologies
  1. Chef Infrastructure

About this book

Irrespective of whether you're a systems administrator or a developer, if you're sick and tired of repetitive manual work and not knowing whether you may dare to reboot your server, it's time for you to get your infrastructure automated.

Chef Infrastructure Automation Cookbook has all the required recipes to configure, deploy, and scale your servers and applications, irrespective of whether you manage 5 servers, 5,000 servers, or 500,000 servers.

Chef Infrastructure Automation Cookbook is a collection of easy-to-follow, step-by-step recipes showing you how to solve real-world automation challenges. Learn techniques from the pros and make sure you get your infrastructure automation project right the first time.

Chef Infrastructure Automation Cookbook takes you on a journey through the many facets of Chef. It teaches you simple techniques as well as fully fledged real-world solutions. By looking at easily digestible examples, you'll be able to grasp the main concepts of Chef, which you'll need for automating your own infrastructure. Instead of wasting time trying to get existing community cookbooks running in your environment, you'll get ready made code examples to get you started.

After describing how to use the basic Chef tools, the book shows you how to troubleshoot your work and explains the Chef language. Then, it shows you how to manage users, applications, and your whole cloud infrastructure. The book concludes by providing you additional, indispensable tools and giving you an in-depth look into the Chef ecosystem.

Chef Infrastructure Automation Cookbook will help you learn the techniques of the pros by walking you through a host of step-by-step guides to solve real-world infrastructure automation challenges.

Publication date:
August 2013
Publisher
Packt
Pages
276
ISBN
9781849519229

 

Chapter 1. Chef Infrastructure

"What made Manhattan Manhattan was the underground infrastructure, that engineering marvel."

- Andrew Cuomo

In this chapter, we will cover the following:

  • Using version control

  • Installing Chef on your workstation

  • Using the Hosted Chef platform

  • Managing virtual machines with Vagrant

  • Creating and using cookbooks

  • Inspecting files on your Chef Server with Knife

  • Defining cookbook dependencies

  • Managing cookbook dependencies with Berkshelf

  • Downloading and integrating cookbooks as vendor branches into your Git repository

  • Using custom Knife plugins

  • Changing organizations based on the current Git branch

  • Deleting a node from the Chef Server

  • Running Chef Solo

  • Using roles

  • Using environments

  • Freezing cookbooks

  • Running Chef Client as a daemon

  • Using the Chef console (Chef Shell)

 

Introduction


This chapter will cover the basics of Chef, including common terminology, workflow practices, and various tools surrounding Chef. We will explore version control using Git, walk through working with community cookbooks, and running those cookbooks on your own servers to configure them the way you need them.

First, let's talk about the terminology used in the Chef universe.

A cookbook is a collection of recipes – codifying the actual resources, which should be installed and configured on your node – and the files and configuration templates needed.

Once you've written your cookbooks, you need a way to deploy them to the nodes you want to provision. Chef offers multiple ways for this task. The most widely used way is to use a central Chef Server . You can either run your own or sign up for Opscode's Hosted Chef.

The Chef Server is the central registry where each node needs to get registered. The Chef Server distributes the cookbooks to the nodes based on their configuration settings.

Knife is Chef's command-line tool called to interact with the Chef Server. You use it for uploading cookbooks and managing other aspects of Chef.

On your nodes, you need to install Chef Client – the part that retrieves the cookbooks from the Chef Server and executes them on the node.

In this chapter, we'll see the basic infrastructure components of your Chef setup at work and learn how to use the basic tools. Let's get started with having a look at how to use Git as a version control system for your cookbooks.

 

Using version control


Do you manually back up every file before you change it? And do you invent creative filename extensions like _me and _you when you try to collaborate on a file? If you answer yes to any of the preceding questions, it's time to rethink your process.

A version control system (VCS) helps you stay sane when dealing with important files and collaborating on them.

Using version control is a fundamental part of any infrastructure automation. There are multiple solutions (some free, some paid) for managing source version control including Git, SVN, Mercurial, and Perforce. Due to its popularity among the Chef community, we will be using Git. However, you could easily use any other version control system with Chef.

Note

Don't even think about building your Infrastructure As Code without using a version control system to manage it!

Getting ready

You'll need Git installed on your box. Either use your operating system's package manager (such as Apt on Ubuntu or Homebrew on OS X), or simply download the installer from www.git-scm.org.

Git is a distributed version control system. This means that you don't necessarily need a central host for storing your repositories. But in practice, using GitHub as your central repository has proven to be very helpful. In this book, I'll assume that you're using GitHub. Therefore, you need to go to github.com and create a (free) account to follow the instructions given in this book. Make sure that you upload your SSH key following the instructions at https://help.github.com/articles/generating-ssh-keys, so that you're able to use the SSH protocol to interact with your GitHub account.

As soon as you've created your GitHub account, you should create your repository by visiting https://github.com/new and using chef-repo as the repository name.

How to do it...

Before you can write any cookbooks, you need to set up your initial Git repository on your development box. Opscode provides an empty Chef repository to get you started. Let's see how you can set up your own Chef repository with Git using Opscode's skeleton.

  1. Download Opscode's skeleton Chef repository as a tarball:

    [email protected] $ wget http://github.com/opscode/chef-repo/tarball/master
    
    ...TRUNCATED OUTPUT...
    2013-07-05 20:54:24 (125 MB/s) - 'master' saved [9302/9302]
  2. Extract the downloaded tarball:

    [email protected] $ tar zvf master
    
  3. Rename the directory. Replace 2c42c6a with whatever your downloaded tarball contained in its name:

    [email protected] $ mv opscode-chef-repo-2c42c6a/ chef-repo
    
  4. Change into your newly created Chef repository:

    [email protected] $ cd chef-repo/
    
  5. Initialize a fresh Git repository:

    [email protected]:~/chef-repo $ git init .
    Initialized empty Git repository in /Users/mma/work/chef-repo/.git/
  6. Connect your local repository to your remote repository on github.com. Make sure to replace mmarschall with your own GitHub username:

    [email protected]:~/chef-repo $ git remote add origin [email protected]:mmarschall/chef-repo.git
    
  7. Add and commit Opscode's default directory structure:

    [email protected]:~/chef-repo $ git add .
    [email protected]:~/chef-repo $ git commit -m "initial commit"
    
    [master (root-commit) 6148b20] initial commit
     10 files changed, 339 insertions(+), 0 deletions(-)
     create mode 100644 .gitignore
    ...TRUNCATED OUTPUT...
    create mode 100644 roles/README.md
  8. Push your initialized repository to GitHub. This makes it available to all your co-workers to collaborate on it.

    [email protected]:~/chef-repo $ git push -u origin master
    
    ...TRUNCATED OUTPUT...
    To [email protected]:mmarschall/chef-repo.git
     * [new branch]      master -> master

How it works...

You've downloaded a tarball containing Opscode's skeleton repository. Then, you've initialized your chef-repo and connected it to your own repository on GitHub.

After that, you've added all the files from the tarball to your repository and committed them. This makes Git track your files and the changes you make later.

As a last step, you've pushed your repository to GitHub, so that your co-workers can use your code too.

There's more...

Let's assume you're working on the same chef-repo repository together with your co-workers. They cloned your repository, added a new cookbook called other_cookbook, committed their changes locally, and pushed their changes back to GitHub. Now it's time for you to get the new cookbook down to your own laptop.

Pull your co-workers, changes from GitHub. This will merge their changes into your local copy of the repository.

[email protected]:~/chef-repo $ git pull
From github.com:mmarschall/chef-repo
 * branch            master     -> FETCH_HEAD
...TRUNCATED OUTPUT...
create mode 100644 cookbooks/other_cookbook/recipes/default.rb

In the case of any conflicting changes, Git will help you merge and resolve them.

See also

 

Installing Chef on your workstation


If you want to use Chef, you'll need to install it on your local workstation first. You'll have to develop your configurations locally and use Chef to distribute them to your Chef Server.

Opscode provides a fully packaged version, which does not have any external prerequisites. This fully packaged Chef is called the Omnibus Installer. We'll see how to use it in this section.

Getting ready

Make sure you've curl installed on your box by following the instructions available at http://curl.haxx.se/download.html.

How to do it...

Let's see how to install Chef on your local workstation using Opscode's Omnibus Chef installer:

  1. In your local shell, run the following command:

    [email protected]:~/chef-repo $ curl -L https://www.opscode.com/chef/install.sh | sudo bash
    
    Downloading Chef...
    ...TRUNCATED OUTPUT...
    Thank you for installing Chef!
  2. Add the newly installed Ruby to your path:

    [email protected]:~ $ echo 'export PATH="/opt/chef/embedded/bin:$PATH"' >> ~/.bash_profile && source ~/.bash_profile
    

How it works...

The Omnibus Installer will download Ruby and all the required Ruby gems into /opt/chef/embedded. By adding the /opt/chef/embedded/bin directory to your .bash_profile, the Chef command-line tools will be available in your shell.

There's more...

If you already have Ruby installed in your box, you can simply install the Chef Ruby gem by running [email protected]:~ $ gem install chef.

See also

 

Using the Hosted Chef platform


If you want to get started with Chef right away (without the need to install your own Chef Server) or want a third party to give you an Service Level Agreement (SLA) for your Chef Server, you can sign up for Hosted Chef by Opscode. Opscode operates Chef as a cloud service. It's quick to set up and gives you full control, using users and groups to control the access permissions to your Chef setup. We'll configure Knife, Chef's command-line tool to interact with Hosted Chef, so that you can start managing your nodes.

Getting ready

Before being able to use Hosted Chef, you need to sign up for the service. There is a free account for up to five nodes.

Visit http://www.opscode.com/hosted-chef and register for a free trial or the free account.

I registered as the user webops with an organization short-name of awo.

After registering your account, it is time to prepare your organization to be used with your chef-repo repository.

How to do it...

Carry out the following steps to interact with the Hosted Chef:

  1. Navigate to http://manage.opscode.com/organizations. After logging in, you can start downloading your validation keys and configuration file.

  2. Select your organization to be able to see its contents using the web UI.

  3. Regenerate the validation key for your organization and save it as <your-organization-short-name>.pem in the .chef directory inside your chef-repo repository.

  4. Generate the Knife config and put the downloaded knife.rb into the .chef directory inside your chef-repo directory as well. Make sure you replace webops with the username you chose for Hosted Chef and awo with the short-name you chose for your organization:

    current_dir = File.dirname(__FILE__)
    log_level                :info
    log_location             STDOUT
    node_name                "webops"
    client_key               "#{current_dir}/webops.pem"
    validation_client_name   "awo-validator"
    validation_key           "#{current_dir}/awo-validator.pem"
    chef_server_url          "https://api.opscode.com/organizations/awo"
    cache_type               'BasicFile'
    cache_options( :path => "#{ENV['HOME']}/.chef/checksums" ) 
    cookbook_path            ["#{current_dir}/../cookbooks"]
  5. Use Knife to verify that you can connect to your hosted Chef organization. It should only have your validator client so far. Instead of awo, you'll see your organization's short-name:

    [email protected]:~/chef-repo $ knife client list
    awo-validator

How it works...

Hosted Chef uses two private keys (called validators): one for the organization and the other for every user. You need to tell Knife where it can find these two keys in your knife.rb file.

The following two lines of code in your knife.rb file tells Knife about which organization to use and where to find its private key:

validation_client_name   "awo-validator"
validation_key           "#{current_dir}/awo-validator.pem"

The following line of code in your knife.rb file tells Knife about where to find your users' private key:

client_key               "#{current_dir}/webops.pem"

And the following line of code in your knife.rb file tells Knife that you're using Hosted Chef. You will find your organization name as the last part of the URL:

chef_server_url          "https://api.opscode.com/organizations/awo"

Using the knife.rb file and your two validators Knife can now connect to your organization hosted by Opscode.

You do not need your own, self-hosted Chef Server, nor do you need to use Chef Solo in this setup.

There's more...

This setup is good for you if you do not want to worry about running, scaling, and updating your own Chef Server and if you're happy with saving all your configuration data in the cloud (under Opscode's control).

If you need to have all your configuration data within your own network boundaries, you might sign up for Private Chef, which is a fully supported and enterprise-ready version of Chef Server.

If you don't need any advanced enterprise features like role-based access control or multi-tenancy, then the open source version of Chef Server might be just right for you.

See also

 

Managing virtual machines with Vagrant


Developing Chef cookbooks requires you to run your work-in-progress cookbooks multiple times on your nodes. To make sure they work, you need a clean, initial state of your nodes every time you run them. You can achieve this by using Virtual Machines (VM). But manually setting up and destroying VMs is tedious and breaks your development flow.

Vagrant is a command-line tool that provides you with a configurable, reproducible, and portable development environment by enabling you to manage VMs. It lets you define and use preconfigured disk images to create new VMs. Also, you can configure Vagrant to use provisioners such as Shell scripts, Puppet, or Chef to bring your VM into the desired state.

In this recipe, we will see how to use Vagrant to manage VMs using VirtualBox and Chef Client as the provisioner.

Getting ready

Download and install VirtualBox at https://www.virtualbox.org/wiki/Downloads.

Download and install Vagrant at http://downloads.vagrantup.com/.

Install the Vagrant Omnibus plugin to enable Vagrant to install Chef Client on your VM by running the following commands:

[email protected]:~/chef-repo $ vagrant plugin install vagrant-omnibus
Installing the 'vagrant-omnibus' plugin. This can take a few minutes...
Installed the plugin 'vagrant-omnibus (1.1.0)'!

How to do it...

Let's create and boot a virtual node by using Vagrant:

  1. Visit https://github.com/opscode/bento and choose a Vagrant box for basing your VMs on. We'll use opscode-ubuntu-12.04 in this example.

  2. The URL of the opscode-ubuntu-12.04 box is https://opscode-vm-bento.s3.amazonaws.com/vagrant/opscode_ubuntu-12.04_provisionerless.box.

  3. Edit your new Vagrantfile. Make sure that you replace <YOUR-ORG> with the name of your organization on the Chef Server. Use the name and URL of the box file you noted down in the first step as config.vm.box and config.vm.box_url:

    [email protected]:~/chef-repo $ subl Vagrantfile
    
    Vagrant.configure("2") do |config|
      config.vm.box = "opscode-ubuntu-12.04"
      config.vm.box_url = https://opscode-vm-bento.s3.amazonaws.com/vagrant/opscode_ubuntu-12.04_provisionerless.box
      config.omnibus.chef_version = :latest
    
      config.vm.provision :chef_client do |chef|
        chef.provisioning_path = "/etc/chef"
        chef.chef_server_url = "https://api.opscode.com/organizations/<YOUR_ORG>"
        chef.validation_key_path = "/.chef/<YOUR_ORG>-validator.pem"
        chef.validation_client_name = "<YOUR_ORG>-validator"
        chef.node_name = "server"
      end
    end 
  4. Create your virtual node using Vagrant:

    [email protected]:~/chef-repo $ vagrant up
    
    Bringing machine 'server' up with 'virtualbox' provider...
    ...TRUNCATED OUTPUT...
    [server] Importing base box 'opscode-ubuntu-12.04'...
    ...TRUNCATED OUTPUT...
    [server] Installing Chef 11.4.4 Omnibus package...
    [server] Running provisioner: chef_client...
    Creating folder to hold client key...
    Uploading chef client validation key...
    Generating chef JSON and uploading...
    Running chef-client...
     [2013-05-27T20:06:04+00:00] INFO: *** Chef 11.4.4 ***
    ...TRUNCATED OUTPUT...
  5. Log in to your virtual node using SSH:

    [email protected]:~/chef-repo $ vagrant ssh
    
    Welcome to Ubuntu 12.04.2 LTS (GNU/Linux 3.5.0-23-generic x86_64)
    
     * Documentation:  https://help.ubuntu.com/
    Last login: Wed Apr 24 07:30:09 2013 from 10.0.2.2
    [email protected]:~$

How it works...

The Vagrantfile is written in a Ruby Domain Specific Language (DSL) for configuring the Vagrant virtual machines. We want to boot a simple Ubuntu VM. Let's go through the Vagrantfile step-by-step.

First, we create a config object. Vagrant will use this config object to configure the VM:

Vagrant.configure("2") do |config|
  ...
end

Inside the config block, we tell Vagrant which VM image to use, in order to boot the node:

  config.vm.box = "opscode-ubuntu-12.04"
  config.vm.box_url = "https://opscode-vm-bento.s3.amazonaws.com/vagrant/opscode_ubuntu-12.04_provisionerless.box"

We want to boot our VM using a so-called Bento Box provided by Opscode. We use Ubuntu version 12.04 here.

Note

If you have never used the box before, Vagrant will download the image file (a few hundred megabytes) when you run vagrant up for the first time.

As we want our VM to have Chef Client installed, we tell the Vagrant Omnibus plugin to use the latest version of Chef Client:

  config.omnibus.chef_version = :latest

After selecting the VM image to boot, we configure how to provision the box using Chef. The Chef configuration happens in a nested Ruby block:

  config.vm.provision :chef_client do |chef|
  ...
  end

Inside this chef block, we need to instruct Vagrant on how to hook up our virtual node to the Chef Server. First, we need to tell Vagrant where to store all the Chef stuff on your node:

    chef.provisioning_path = "/etc/chef"

Vagrant needs to know the API endpoint of your Chef Server. If you use Hosted Chef, it is https://api.opscode.com/organizations/<YOUR_ORG>. You need to replace <YOUR_ORG> with the name of the organization you created in your account on Hosted Chef. If you are using your own Chef Server, change the URL accordingly:

    chef.chef_server_url = "https://api.opscode.com/organizations/<YOUR_ORG>"

While creating your organization on Hosted Chef, you must have downloaded your private key. Tell Vagrant where to find this file:

    chef.validation_key_path = /.chef/<YOUR_ORG>—validator.pem"

Also, you need to tell Vagrant as which client it should validate itself against the Chef Server:

    chef.validation_client_name = "<YOUR_ORG>-validator"

Finally, you should tell Vagrant how to name your node:

    chef.node_name = "server"

After configuring your Vagrantfile, all you need to do is run the basic Vagrant commands like vagrant up, vagrant provision, and vagrant ssh. To stop your VM, just run the vagrant halt command.

There's more...

If you want to start from scratch again, you will have to destroy your VM as well as delete both the client and the node from your Chef Server by running the following commands:

[email protected]:~/chef-repo $ vagrant destroy
[email protected]:~/chef-repo $ knife node delete server -y && knife client delete server -y

Alternatively, you may use the Vagrant Butcher plugin found at https://github.com/cassianoleal/vagrant-butcher.

See also

 

Creating and using cookbooks


Cookbooks are an essential part of Chef. You can easily create them using Knife, Chef's command-line tool. In this section (and many of the following sections), I will assume that you're using a Chef Server to manage your infrastructure. You can either set up your own or use the Hosted Chef as described previously.

In this section, we'll create and apply a simple cookbook using Knife.

Getting ready

Make sure you've Chef installed and a node available for testing. Check out the installation instructions at http://learnchef.com if you need help here.

Edit your knife.rb file and add the following three lines to it, filling in your own values:

cookbook_copyright "your company"
cookbook_license "apachev2"
cookbook_email "your email address"

Note

The Apache 2 license is the most commonly found in cookbooks, but you're free to choose whichever suits your needs. If you put none as the cookbook_license, Knife will put "All rights reserved" into your recipe's metadata file.

Knife will use the preceding values as default whenever you create a new cookbook.

How to do it...

Carry out the following steps to create and use cookbooks:

  1. Create a cookbook by running the following with the name my_cookbook:

    [email protected]:~/chef-repo $ knife cookbook create my_cookbook
    
    ** Creating cookbook my_cookbook
    ** Creating README for cookbook: my_cookbook
    ** Creating CHANGELOG for cookbook: my_cookbook
    ** Creating metadata for cookbook: my_cookbook
  2. Upload your new cookbook to the Chef Server:

    [email protected]:~/chef-repo $ knife cookbook upload my_cookbook
    
    Uploading my_cookbook    [0.1.0]
    Uploaded 1 cookbook.
  3. Add the your node's run list. In this example, the name of the node is server:

    [email protected]:~/chef-repo $ knife node run_list add server recipe[my_cookbook]
    
    server:
      run_list: recipe[my_cookbook]
  4. Run Chef Client on your node:

    [email protected]:~$ sudo chef-client
    

How it works...

Knife is the command-line interface for the Chef Server. It uses the RESTful API exposed by the Chef Server to do its work and helps you to interact with the Chef Server.

The knife command supports a host of commands structured like the following:

knife <subject> <command>

The <subject> used in this section is either a cookbook or a node. The commands we use are create or upload for the cookbook, and run_list add for the node.

See also

  • The Using the Hosted Chef platform section

 

Inspecting files on your Chef Server with Knife


Sometimes, you may want to peek into the files stored on your Chef Server. You might not be sure about an implementation detail of that specific cookbook version, which is currently installed on your Chef Server, and would want to look it up. Knife can help you out by letting you show various aspects of the files stored on your Chef Server.

Getting ready

Make sure you have the iptables cookbook installed locally and uploaded to your Chef Server.

  1. Install the iptables community cookbook by executing the following command:

    [email protected]:~/work/chef_helpster $ knife cookbook site install iptables
    
    Installing iptables to /Users/mma/work/chef-repo/cookbooks
    ...TRUNCATED OUTPUT...
  2. Upload the iptables cookbook to your Chef Server by executing the following command:

    [email protected]:~/work/chef_helpster $ knife cookbook
    
    Uploading iptables       [0.12.0]
    Uploaded 1 cookbook.

How to do it...

Let's find out how Knife can help you to look into a cookbook stored on your Chef Server:

  1. First, you want to find out the current version of the cookbook you're interested in. In our case, we're interested in the iptables cookbook:

    [email protected]:~/work/chef_helpster $ knife cookbook show iptables
    iptables   0.12.0
  2. Then, you can look up the definitions of the iptables cookbook:

    [email protected]:~/work/chef_helpster $ knife cookbook show iptables 0.12.0 definitions
    
      checksum:     189188109499d68612a5b95b6809b580
      name:         iptables_rule.rb
      path:         definitions/iptables_rule.rb
      specificity:  default
      url:          https://s3.amazonaws.com/opscode-platform...
  3. Now, you can even show the contents of the iptables_rule.rb definition file as stored on the Chef Server:

    m[email protected]:~/work/chef_helpster $ knife cookbook show iptables 0.12.0 definitions iptables_rule.rb
    
    #
    # Cookbook Name:: iptables
    # Definition:: iptables_rule
    #
    #
    define :iptables_rule, :enable => true, :source => nil, :variables => {} do
    ...TRUNCATED OUTPUT...
    end

How it works...

The knife show sub-command helps you understand what exactly is stored on the Chef Server. It let's you drill down into specific sections of your cookbooks and see the exact content of the files stored on your Chef Server.

There's more...

Using Chef 11, you can pass patterns to the knife show command to tell it what exactly you want to see. Showing the contents of the iptables_rule definition can be done like this, in addition to the way we used previously:

[email protected]:~/work/chef_helpster $ knife show cookbooks/iptables/definitions/*
cookbooks/iptables/definitions/iptables_rule.rb:
#
# Cookbook Name:: iptables
# Definition:: iptables_rule
#
#
define :iptables_rule, :enable => true, :source => nil, :variables => {} do
...TRUNCATED OUTPUT...
end

See also

 

Defining cookbook dependencies


Quite often, you might want to use features of other cookbooks in your own cookbooks. For example, if you want to make sure that all packages required for compiling the C software are installed, you might want to include the build-essential cookbook that does just that. When using Chef Server, it needs to know about such dependencies in your cookbooks. You need to declare them in the cookbook's metadata.

Getting ready

Make sure you've a cookbook named my_cookbook and the run_list command of your node includes my_cookbook, as described in the Creating and using cookbooks recipe.

How to do it...

Edit the metadata of your cookbook in the file cookbooks/my_cookbook/metadata.rb to add a dependency to the build-essential cookbook:

[email protected]:~/chef-repo $ subl cookbooks/my_cookbook/metadata.rb
...
depends 'build-essential'
depends 'apache2', '>= 1.0.4'

How it works...

If you want to use a feature of another cookbook inside your cookbook, you will need to include the other cookbook in your recipe.

include_recipe 'build-essential'

To tell the Chef Server that your cookbook requires the build-essential cookbook, you need to declare that dependency in the metadata.rb file. If you've uploaded all the dependencies on your Chef Server, the Chef Server will then send all the required cookbooks to the node.

Tip

Declaring dependencies is not necessary, if you're using Chef Solo.

The first depends call tells the Chef Server that your cookbook depends on the latest version of the build-essential cookbook.

The second depends call tells the Chef Server that your cookbook depends on a version of the apache2 cookbook, which is greater or equal to the version 1.0.4. You may use any of these version constraints with your depends calls:

  • < (less than)

  • <= (less than or equal to)

  • = (equal to)

  • >= (greater than or equal to)

  • ~> (approximately greater than)

  • > (greater than)

There's more...

If you're using the foodcritic gem and include another recipe inside your recipe, without declaring the cookbook dependency in your metadata.rb file, foodcritic will warn you:

[email protected]:~/chef-repo $ foodcritic my_cookbook
FC007: Ensure recipe dependencies are reflected in cookbook metadata: cookbooks/my_cookbook/recipes/default.rb:9

Additionally, you can declare conflicting cookbooks through the conflicts call:

conflicts "nginx"

Of course, you can use version constraints exactly the way you did with depends.

See also

  • The Inspecting files on your Chef Server with Knife section

  • Find out how to use foodcritic in the Flagging problems in your Chef cookbooks section in Chapter 2, Evaluating and Troubleshooting Cookbooks and Chef Runs

 

Managing cookbook dependencies with Berkshelf


It's a pain to manually ensure that you've installed all the cookbooks, which another cookbook depends on. You've to download each and every one of them manually only to find out that with each downloaded cookbook, you inherit another set of dependent cookbooks.

And even if you use knife cookbook site install, which installs all the dependencies locally for you, your cookbook directory and your repository get cluttered with all those cookbooks. Usually, you don't really care about all those cookbooks and don't want to see or even manage them.

This is where Berkshelf comes into play. It works like Bundler for Ruby gems, managing cookbook dependencies for you. It downloads all the defined dependencies recursively.

Instead of polluting your Chef repository, it stores all the cookbooks in a central location. You just commit your Berkshelf dependency file (called Berksfile) to your repository, and every colleague or build server can download and install all those dependent cookbooks based on it.

Let's see how to use Berkshelf to manage the dependencies of your cookbook.

Getting ready

Make sure you've a cookbook named my_cookbook and the run_list of your node includes my_cookbook as described in the Creating and using cookbooks section.

How to do it...

Berkshelf helps you to keep those utility cookbooks out of your Chef repository. This makes it much easier to maintain the cookbooks, which really matter.

Let's see how to write a cookbook running a bunch of utility recipes and manage the required cookbooks with Berkshelf:

  1. Create a Gemfile containing the berkshelf gem:

    [email protected]:~/chef-repo $ subl Gemfile
    
    source 'https://rubygems.org'
    gem 'berkshelf'
  2. Run Bundler to install the gem:

    [email protected]:~/chef-repo $ bundler install
    
    Fetching gem metadata from https://rubygems.org/
    ...TRUNCATED OUTPUT...
    Installing berkshelf (2.0.7)
    Using bundler (1.3.5)
    Your bundle is complete!
  3. Edit your cookbook's metadata:

    [email protected]:~/chef-repo $ subl cookbooks/my_cookbook/metadata.rb
    
    ...
    depends "chef-client"
    depends "apt"
    depends "ntp"
  4. Edit your cookbook's default recipe:

    [email protected]:~/chef-repo $ subl cookbooks/my_cookbook/recipes/default.rb
    
    ...
    include_recipe "chef-client"
    include_recipe "apt"
    include_recipe "ntp"
  5. Create your Berksfile:

    [email protected]:~/chef-repo $ subl Berksfile
    
    site :opscode
    
    metadata
  6. Run Berkshelf to install all the required cookbooks:

    [email protected]:~/chef-repo $ cd cookbooks/my_cookbook
    [email protected]:~/chef-repo/cookbooks/my_cookbook $ berks install
    
    Using my_cookbook (0.1.0) from metadata
    Installing chef-client (3.0.4) from site: 'http://cookbooks.opscode.com/api/v1/cookbooks'
    Installing cron (1.2.4) from site: 'http://cookbooks.opscode.com/api/v1/cookbooks'
    Installing apt (2.0.0) from site: 'http://cookbooks.opscode.com/api/v1/cookbooks'
    Installing ntp (1.3.2) from site: 'http://cookbooks.opscode.com/api/v1/cookbooks'
  7. Upload all the cookbooks on the Chef Server:

    [email protected]:~/chef-repo/cookbooks/my_cookbook $ berks upload
    
    Using my_cookbook (0.1.0)
    ...TRUNCATED OUTPUT...
    Uploading ntp (1.3.2) to: 'https://api.opscode.com:443/organizations/agilewebops'

How it works...

Berkshelf comes as a Ruby gem, which we need to install first.

Then, we create our cookbook and tell it to use a few other cookbooks.

Instead of manually installing all the cookbooks using knife cookbook site install, we create a Berksfile besides the metadata.rb file.

The Berksfile is pretty simple. We tell Berkshelf to use the Opscode community site as the default source for all cookbooks:

site :opscode

And we tell Berkshelf to read the metadata.rb file to find all the required cookbooks. This is the simplest way when working inside a single cookbook. Please see the following There's more… section to find an example of a more advanced usage of the Berksfile.

After telling Berkshelf where to find all the required cookbook names, we use it to install all those cookbooks:

berks install

Berkshelf stores cookbooks in ~/.berkshelf/cookbooks by default. This keeps your Chef repository clutter free. Instead of having to manage all the required cookbooks inside your own Chef repository, Berkshelf takes care of them. You simply need to check in Berksfile with your cookbook, and everyone using your cookbook can download all the required cookbooks using Berkshelf.

To make sure that there's no mix-up with different cookbook versions when sharing your cookbook, Berkshelf creates a file called Berksfile.lock alongside Berksfile. There you'll find the exact versions of all the cookbooks that Berkshelf installed:

{
  "sha": "b7d5bda18ccfaffe88a7b547420c670b8f922ff1",
  "sources": {
    "my_cookbook": {
      "path": "."
    },
    "chef-client": {
      "locked_version": "3.0.4"
    },
    "cron": {
      "locked_version": "1.2.4"
    },
    "apt": {
      "locked_version": "2.0.0"
    },
    "ntp": {
      "locked_version": "1.3.2"
    }
  }
}

Berkshelf will only use the exact versions specified in the Berksfile.lock file, if it finds this file.

Finally, we use Berkshelf to upload all the required cookbooks on the Chef Server:

berks upload

There's more...

Berkshelf integrates tightly with Vagrant via the vagrant-berkshelf plugin. You can set up Berkshelf and Vagrant in such a way that Berkshelf installs and uploads all the required cookbooks on your Chef Server whenever you execute vagrant up or vagrant provision. You'll save all the work of running berks install and berks upload manually before creating your node with Vagrant.

Let's see how you can integrate Berkshelf and Vagrant.

First, you need to install the Berkshelf plugin for Vagrant:

[email protected]:~/work/chef-repo (master)$ vagrant plugin install vagrant-berkshelf
Installing the 'vagrant-berkshelf' plugin. This can take a few minutes...
Installed the plugin 'vagrant-berkshelf (1.3.2)'!

Then, you need to tell Vagrant that you want to use the plugin. You do this by enabling the plugin in your Vagrantfile:

[email protected]:~/work/chef-repo (master)$ subl Vagrantfile
...
config.berkshelf.enabled = true
...

Then, you need a Berksfile in the root directory of your Chef repository, to tell Berkshelf which cookbooks to install on each Vagrant run:

cookbook 'my_cookbook', path: 'cookbooks/my_cookbook'

Eventually, you can start your VM using Vagrant. Berkshelf will first download and install all the required cookbooks in the Berkshelf, and upload them to the Chef Server. Only after all the cookbooks are made available on the Chef Server by Berkshelf, will Vagrant go on:

[email protected]:~/work/chef-repo $ vagrant up
Bringing machine 'server' up with 'virtualbox' provider...
...TRUNCATED OUTPUT...
 [Berkshelf] Uploading cookbooks to 'https://api.opscode.com/organizations/agilewebops'
...TRUNCATED OUTPUT...

This way, using Berkshelf together with Vagrant, you save a lot of manual steps and get faster cycle times for your cookbook development.

See also

 

Downloading and integrating cookbooks as vendor branches into your Git repository


The Opscode community offers a wide variety of ready-made cookbooks for many major software packages. They're a great starting point for your own infrastructure. But, usually you need to modify these cookbooks to suit your needs. Modifying your local copy of a community cookbook leaves you in the dilemma of not being able to update to the latest version of the community cookbook without losing your local changes.

Getting ready

You'll need to make sure that your local Git repository is clean and does not have any uncommitted changes:

[email protected]:~/chef-repo $ git status
# On branch master
nothing to commit (working directory clean)

How to do it...

Carry out the following steps:

  1. Go to http://community.opscode.com/cookbooks and search for the cookbook you need. In our example, we will use the mysql cookbook, which is featured right there on top under the All Categories list as well as above the Databases section. All we need is to note down the exact name of the cookbook in this case it's simply mysql.

  2. Use Knife to pull down the cookbook and to integrate it with your local repository:

    [email protected]:~/chef-repo $ knife cookbook site install mysql
    
    Installing mysql to /Users/mma/work/chef-repo/cookbooks
    …TRUNCATED OUTPUT…
    Cookbook build-essential version 1.2.0 successfully installed
  3. Verify the downloaded cookbooks:

    [email protected]:~/chef-repo $ cd cookbooks
    [email protected]:~/chef-repo/cookbooks $ ls -l
    
    total 8
    -rw-r--r--   1 mma  staff  3064 23 Nov 22:02 README.md
    drwxr-xr-x  12 mma  staff   408 28 Nov 20:40 build-essential
    drwxr-xr-x  13 mma  staff   442 28 Nov 20:34 my_cookbook
    drwxr-xr-x  15 mma  staff   510 28 Nov 20:39 mysql
    drwxr-xr-x   7 mma  staff   238 28 Nov 20:39 openssl
  4. Validate the Git status:

    [email protected]:~/chef-repo/cookbooks $ git status
    
    # On branch master
    # Your branch is ahead of 'origin/master' by 3 commits.
    #
    nothing to commit (working directory clean)
  5. You might have noticed that your local branch has received three commits. Let's have a look at those:

    [email protected]:~/chef-repo/cookbooks $ git log
    
    commit 766bd4098184f4d188c75daa49e12abb5b1fd360
    Author: Matthias Marschall <[email protected]>
    Date:   Wed Nov 28 20:40:01 2012 +0100
    commit 766bd4098184f4d188c75daa49e12abb5b1fd360
    Author: Matthias Marschall <[email protected]>
    Date:   Wed Nov 28 20:40:01 2012 +0100
    
        Import build-essential version 1.2.0
    
    commit 6ad70f1fbbb96df1fc55c3237966c60d156d6026
    Author: Matthias Marschall <[email protected]>
    Date:   Wed Nov 28 20:39:59 2012 +0100
    
        Import openssl version 1.0.0
    
    commit d03dd06f3c931078c2a9943a493955780e39bf22
    Author: Matthias Marschall <[email protected]>
    Date:   Wed Nov 28 20:39:58 2012 +0100
    
        Import mysql version 2.0.2

The knife command successfully downloaded and imported the mysql cookbook as well as its dependencies: the build-essential and openssl cookbooks.

How it works...

Knife executes a set of commands to download the desired cookbook and to integrate it with your local repository.

Let's have a look at the output of the knife cookbook site install command again and go through it step-by-step.

First, the command makes sure that you're on the master branch of your repository:

Checking out the master branch.

The next step is to create a new vendor branch for the mysql cookbook if none exists so far:

Creating pristine copy branch chef-vendor-mysql.

Then it downloads the tarball, removes any older version, uncompresses the new tarball, and removes it after successfully extracting its contents into a new cookbook directory:

Downloading mysql from the cookbooks site at version 2.0.2 to /Users/mma/work/chef-repo/cookbooks/mysql.tar.gz
Cookbook saved: /Users/mma/work/chef-repo/cookbooks/mysql.tar.gz
Removing pre-existing version.
Uncompressing mysql version 2.0.2.
Removing downloaded tarball

Now, it's time to commit the newly extracted files to the vendor branch:

1 files updated, committing changes

Finally, it tags it with the current version of the cookbook:

Creating tag cookbook-site-imported-mysql-2.0.2

The knife cookbook site install command executes all the previous mentioned steps for all the cookbooks the desired cookbook depends on, by default.

Eventually, you end up with a separate branch, the so-called vendor branch, for every downloaded cookbook integrated into your master branch and nicely tagged. This approach enables you to change whatever you like in your master branch and still pull down newer versions of the community cookbook. Git will automatically merge both the versions or will ask you to remove conflicts manually; all the standard Git procedures.

Tip

Downloading the example code

You can download the example code files for all Packt books you have purchased from your account at http://www.packtpub.com. If you purchased this book elsewhere, you can visit http://www.packtpub.com/support and register to have the files e-mailed directly to you.

There's more...

If you want to integrate the desired cookbook into another branch, use the --branch BRANCH_NAME parameter.

[email protected]:~/chef-repo [experimental] $ knife cookbook site install mysql –-branch experimental
Installing mysql to /Users/mma/work/chef-repo/cookbooks
Checking out the experimental branch.
Pristine copy branch (chef-vendor-mysql) exists, switching to it.
Downloading mysql from the cookbooks site at version 2.0.2 to /Users/mma/work/chef-repo/cookbooks/mysql.tar.gz
Cookbook saved: /Users/mma/work/chef-repo/cookbooks/mysql.tar.gz
Removing pre-existing version.
Uncompressing mysql version 2.0.2.
removing downloaded tarball
No changes made to mysql
Checking out the experimental branch.
…TRUNCATED OUTPUT…

As you can see, instead of checking out the master branch, the knife cookbook site install command uses the experimental branch now.

You can use the -D switch when running the command to avoid downloading all the cookbooks your desired cookbook depends on.

[email protected]:~/chef-repo $ knife cookbook site install mysql -D
Installing mysql to /Users/mma/work/chef-repo/cookbooks
Checking out the master branch.
Pristine copy branch (chef-vendor-mysql) exists, switching to it.
Downloading mysql from the cookbooks site at version 2.0.2 to /Users/mma/work/chef-repo/cookbooks/mysql.tar.gz
Cookbook saved: /Users/mma/work/chef-repo/cookbooks/mysql.tar.gz
Removing pre-existing version.
Uncompressing mysql version 2.0.2.
removing downloaded tarball
No changes made to mysql
Checking out the master branch.

You see that the command stopped after dealing with the mysql cookbook. It did not get the other cookbooks yet.

See also

  • You can use Berkshelf to manage cookbooks and their dependencies for you, which makes the preceding approach obsolete. See the Managing cookbook dependencies with Berkshelf section.

 

Using custom Knife plugins


Knife comes with a set of commands out of the box. The built-in commands deal with the basic elements of Chef like cookbooks, roles, data bags, and so on. But, it would be nice to use Knife for more than just the basic stuff. Fortunately, Knife comes with a plugin API, and there are already a host of useful Knife plugins built by Opscode and the Chef community.

Getting ready

Make sure that you've Bundler installed on your local workstation:

[email protected]:~/chef-repo $ gem install bundler
Fetching: bundler-1.3.5.gem (100%)
Successfully installed bundler-1.3.5
1 gem installed

Make sure you've got an account at Amazon AWS if you want to follow along and try out the knife-ec2 plugin. There are Knife plugins available for most cloud providers. Go through the There's more... section of this section for the list.

How to do it...

Let's see which Knife plugins are available, and try to use one for managing Amazon EC2 instances:

  1. List the Knife plugins that are shipped as Ruby gems:

    [email protected]:~/chef-repo $ gem search -r knife-
    
    *** REMOTE GEMS ***
    knife-audit (0.2.0)
    knife-azure (1.0.2)
    ...TRUNCATED OUTPUT...
    knife-ec2 (0.6.4)
    ...TRUNCATED OUTPUT...
  2. Create a Gemfile containing the EC2 plugin:

    [email protected]:~/chef-repo $ subl Gemfile
    
    source 'https://rubygems.org'
    gem 'knife-ec2', '~>0.6.4'
  3. Install the EC2 plugin for managing servers in the Amazon AWS cloud:

    [email protected]:~/chef-repo $ bundle install
    
    Fetching gem metadata from https://rubygems.org/
    ...TRUNCATED OUTPUT...
    Installing knife-ec2 (0.6.4)
    Using bundler (1.3.5)
    Your bundle is complete!
  4. List all the available instance types in AWS using the knife ec2 plugin. Please use your own AWS credentials instead of XXX and YYYYY:

    [email protected]:~/chef-repo $ knife ec2 flavor list --aws-access-key-id XXX --aws-secret-access-key YYYYY
    
    ID           Name                                 Arch    RAM     Disk     Cores      
    c1.medium    High-CPU Medium                          32-bit  1740.8  350 GB   5          
    …TRUNCATED OUTPUT…
    m2.xlarge    High-Memory Extra Large                  64-bit  17510.  420 GB   6.5        
    t1.micro     Micro Instance                           0-bit   613     0 GB     2 

How it works...

Knife looks for plugins at various places.

First, it looks into the .chef directory located inside your current Chef repository, to find the plugins specific to this repository:

./.chef/plugins/knife/

Then, it looks into the .chef directory located in your home directory, to find the plugins that you want to use in all your Chef repositories:

~/.chef/plugins/knife/

Finally, it looks for installed gems. Knife will load all the code from any chef/knife/ directory found in your installed Ruby gems. This is the most common way of using plugins developed by Opscode or the Chef community.

There's more...

There are Knife plugins for most of the major cloud providers as well as for most of the major virtualization technologies.

At the time of the writing of this book, the following cloud providers were supported by Knife plugins:

  • Microsoft Azure

  • BlueBox

  • Brightbox

  • Amazon EC2

  • Eucalyptus

  • HP Cloud Services

  • OpenStack

  • Rackspace Cloud

  • Terremark

  • VSphere

  • Apache CloudStack

Virtualization technologies supported by Knife plugins are listed as follows:

  • KVM

  • VMware ESX

  • Vagrant

  • Xenserver

See also

 

Changing organizations based on the current Git branch


Chef has this notion of environments to separate, for example, a staging environment from a production environment. You can define specific cookbook versions to be used only in a specific environment and a few more things.

But for development, you might want to give everyone a separate organization on Hosted Chef, to make sure that no one is stepping on one another's toes while doing heavy refactoring. This is not possible by solely using the environments feature.

Note

Please note that this is not a condoned behavior and has proven to be difficult to manage. It fails for many companies supported directly by Opscode. But, if this is the way to go for you, here you'll learn how.

If you're using separate organizations for each developer, you can automate choosing the right organization, by making your knife.rb aware of your current Git branch. I assume that you use the master branch for maintaining your production-ready cookbooks and the development branch for playing around with your stuff.

Let's see how to let Knife autoselect the correct organization.

Getting ready

Additionally to your default organization in your Hosted Chef account, you need to create a new organization for every totally sandboxed environment.

  1. Create a new organization called YOUR_ORG‑development, for example, awo-development, using the Opscode management console at http://manage.opscode.com.

  2. Create a separate Git branch named development:

    [email protected]:~/chef-repo $ git checkout -b development
    [email protected]:~/chef-repo $ subl Gemfile
    
    ...
    gem 'grit'
  3. Run Bundler to install the Grit gem:

    [email protected]:~/chef-repo $ bundle install
    
    ...TRUNCATED OUTPUT...
    Installing grit (2.5.0)

How to do it...

Let's create a knife.rb file, which evaluates your current Git branch and switches the Hosted Chef organization accordingly.

  1. Put the following lines at the top of your knife.rb file. Replace "awo" with the value you used for YOUR_ORG while getting ready:

    organization_base_name = "awo"
    require 'grit'
    repository = Grit::Repo.new(Dir.pwd)
    current_branch = Grit::Head.current(repository).name
    organization = organization_base_name 
    organization << "-#{current_branch}" unless current_branch == 'master'
  2. Make sure that you set the chef_server_url correctly:

    chef_server_url          "https://api.opscode.com/organizations/#{organization}"
  3. Run the knife command off your Git master branch, replacing "awo" with your chosen short-name for your organization:

    [email protected]:~/chef-repo $ knife node list
    awo
  4. Switch to your development branch:

    [email protected]:~/chef-repo $ git checkout development
    
  5. And, run the knife command again:

    [email protected]:~/chef-repo [development]$ knife node list
    
    awo-development

How it works...

To be able to use grit for getting the current branch name, we require the grit gem.

Next, we instantiate a Grit::Repo object from the current working directory. We then use this Grit::Repo object to retrieve the current branch. From the current branch, simply take the name and store it in the current_branch variable.

Now, it's time to set our organization name to the name of our default organization.

After that, we amend the organization name with a - symbol along with the branch name, unless the branch name equals master. This means that if we're currently in the master branch, Knife will use our default organization (without any suffix). If it is on a git branch, it will attach the suffix -branch_name to our organization name.

Further down, we use the constructed organization name to connect to the Chef Server by calling chef_server_url:

chef_server_url          "https://api.opscode.com/organizations/#{organization}"

There's more...

Your knife.rb file is a plain Ruby file. You can put any Ruby code inside it using any gems you want.

To be a little more flexible, we made our knife.rb file even read an environment variable, CHEF_ORG, which overrides the git branch magic:

organization = ENV['CHEF_ORG'] || begin
  require 'grit'
  repository = Grit::Repo.new(Dir.pwd)
  current_branch = Grit::Head.current(repository).name
  chef_org = "awo" 
  chef_org << "-#{current_branch}" unless current_branch == 'master'
  chef_org
end

As long as you don't set the environment variable CHEF_ORG, everything works as before. But if you call Knife in the following manner, it will use the given environment variable as the organization name directly.

[email protected]:~/chef-repo $ CHEF_ORG=experimental knife node list
experimental

See also

  • The Using the Hosted Chef pla tform section

 

Deleting a node from the Chef Server


Bootstrapping a node not only installs Chef on that node but also creates a client object on the Chef Server as well. The client object is used by the Chef Client to authenticate against the Chef Server on each run.

Additionally to registering a client, a node object is created. The node object is the main data structure, which is used by the Chef Client to converge the node to the desired state.

Getting ready

Make sure you've at least one node registered at your Chef Server, which is safe to remove.

How to do it...

Let's delete the node and the client object to completely remove your node from the Chef Server.

  1. Delete the node object:

    [email protected]:~/chef-repo $ knife node delete my_node
    
    Do you really want to delete my_node? (Y/N) y
    Deleted node[my_node]
  2. Delete the client object:

    [email protected]:~/chef-repo $ knife node client my_node
    
    Do you really want to delete my_node? (Y/N) y
    Deleted client[my_node]

How it works...

To keep your Chef Server clean, it's important to not only manage your node objects but also take care of your client objects.

Knife connects to the Chef Server and deletes the node object with the given name using the Chef Server RESTful API.

The same happens while deleting the client object on the Chef Server.

After deleting both the objects, your node is totally removed from the Chef Server. Now, you can reuse the same node name with a new box or virtual machine.

There's more...

It is a bit tedious and error prone when you have to issue two commands. To simplify things, you can use a Knife plugin called playground.

  1. Add the knife-playground plugin to your Gemfile:

    [email protected]:~/chef-repo $ subl Gemfile
    
    ...
    gem 'knife-playground'
  2. Run Bundler to install the Knife plugin:

    [email protected]:~/chef-repo $ bundle install
    
    ...TRUNCATED OUTPUT...
    Installing knife-playground (0.2.2)
  3. Run the knife pg clientnode delete sub-command:

    [email protected]:~/chef-repo $ knife pg clientnode delete my_node
    
    Deleting CLIENT my_node...
    Do you really want to delete my_node? (Y/N) y
    Deleted client[my_node]
    Deleting NODE my_node...
    Do you really want to delete my_node? (Y/N) y
    Deleted node[my_node]

See also

  • The Managing Virtual Machines with Vagrant section

  • The Using the Hosted Chef platform section

 

Running Chef Solo


If running your own Chef Server seems like overkill and you're not comfortable with using Hosted Chef, you can use Chef Solo to execute cookbooks on your server.

Getting ready

Before you're able to run Chef Solo on your servers, you will need to add two files to your local Chef repository: solo.rb and node.json.

The solo.rb file tells Chef Solo where to find the cookbooks, roles, and data bags.

The node.json file sets the run list (and any other node-specific attributes if required).

  1. Create a solo.rb file inside your Chef repository with the following contents:

    current_dir = File.expand_path(File.dirname(__FILE__))
    file_cache_path "#{current_dir}"
    cookbook_path "#{current_dir}/cookbooks"
    role_path "#{current_dir}/roles"
    data_bag_path "#{current_dir}/data_bags"
  2. Add the file to Git:

    [email protected]:~/chef-repo $ git add solo.rb
    
  3. Create a file called node.json inside your Chef repository with the following contents:

    {
      "run_list": [ "recipe[ntp]" ]
    }
  4. You might need to get the ntp cookbook into your Chef repository:

    [email protected]:~/chef-repo $ knife cookbook site install ntp
    
    Installing ntp to /Users/mma/work/chef-repo/cookbooks
    …TRUNCATED OUTPUT…
    Cookbook ntp version 1.3.0 successfully installed
  5. Add the node.json file to Git:

    [email protected]:~/chef-repo $ git add node.json
    
  6. Commit and push your changes to GitHub so that your server will be able to pull them:

    [email protected]:~/chef-repo $ git commit -m "initial setup for Chef Solo"
    [email protected]:~/chef-repo $ git push
    
    Counting objects: 4, done.
    Delta compression using up to 4 threads.
    ...TRUNCATED OUTPUT...
    To [email protected]:mmarschall/chef-repo.git
       b930647..5bcfab6  master -> master

Now you should be ready to install NTP on your server using Chef Solo.

How to do it...

Let's install NTP on your node using Chef Solo:

  1. Log in to your remote server, which you want to provision with Chef Solo.

  2. Clone your Chef repository. Please replace mmarschall with your own GitHub username:

    [email protected]:~$ git clone git://github.com/mmarschall/chef-repo.git
    
  3. Change into your Chef repository:

    [email protected]:~$ cd chef-repo
    
  4. Run Chef Solo to converge the node:

    [email protected]:~/chef-repo$ sudo chef-solo -c solo.rb -j node.json 
    
    [2012-12-08T22:54:13+01:00] INFO: *** Chef 11.0.0 ***
    [2012-12-08T22:54:13+01:00] INFO: Setting the run_list to ["recipe[ntp]"] from JSON
    ...TRUNCATED OUTPUT...
     [2012-12-08T22:54:16+01:00] INFO: Chef Run complete in 2.388374 seconds
    [2012-12-08T22:54:16+01:00] INFO: Running report handlers
    [2012-12-08T22:54:16+01:00] INFO: Report handlers complete

How it works...

solo.rb configures Chef Solo to look for its cookbooks, roles, and data bags inside the current directory: the Chef repository.

Chef Solo takes its node configuration from a JSON file, in our example we simply called it node.json. If you're going to manage multiple servers, you'll need a separate file for each node.

Then, Chef Solo just executes a Chef run based on the configuration data found in solo.rb and node.json.

Note

Chef Solo has limited functionality when compared to a Chef Server:

  • No node data storage

  • No search inside recipes

  • No environments to manage cookbook versions (you could use Git branches instead)

There's more...

Instead of cloning a GitHub repository on your server, you can collect your cookbooks into one file by using tar and make the resulting tarball available via HTTP. Your server can then download the cookbooks tarball if you tell it where the tarball lives, by using the -r parameter to Chef Solo.

To circumvent the limitations of Chef Solo, there exist various other tools such as littl e-chef or knife-solo.

See also

 

Using roles


Roles are the Chef way to group nodes. Typical cases are to have roles for web servers, database servers, and so on.

You can set custom run lists for all the nodes in your roles and override attribute values from within your roles.

Let's see how to create a simple role.

Getting ready

For the following examples, I assume that you have a node named server and that you have at least one cookbook (I'll use the ntp cookbook) registered with your Chef Server.

How to do it...

Let's create a role and see what we can do with it.

  1. Create a role:

    [email protected]:~/chef-repo $ subl roles/web_servers.rb
    
    name "web_servers"
    description "This role contains nodes, which act as web servers"
    run_list "recipe[ntp]"
    default_attributes 'ntp' => {
      'ntpdate' => {
        'disable' => true
      }
    }
  2. Upload the role to the Chef Server:

    [email protected]:~/chef-repo $ knife role from file web_servers.rb
    Updated Role web_servers!
  3. Assign the role to a node called server:

    [email protected]:~/chef-repo $ 
    knife node edit server
    
      "run_list": [
        "role[web_servers]"
      ]
    Saving updated run_list on node server
  4. Run Chef Client:

    [email protected]:~$ sudo chef-client
    
    ...TRUNCATED OUTPUT...
    [2013-07-25T13:28:24+00:00] INFO: Run List is [role[web_servers]]
    [2013-07-25T13:28:24+00:00] INFO: Run List expands to [ntp]
    ...TRUNCATED OUTPUT...

How it works...

You define a role in a Ruby file inside the roles folder of your Chef repository. A role consists of a name and a description attribute. Additionally, a role usually contains a role-specific run list and role-specific attribute settings.

Every node, that has a role in its run list will have the role's run list expanded into its own. This means all the recipes (and roles) that are in the role's run list will be executed on your nodes.

You need to upload your role to your Chef Server using the knife role from file command.

Only then can you add the role to your node's run list.

Running Chef Client on a node having your role in its run list will execute all the recipes listed in the role.

The attributes you define in your role will be merged with attributes from environments and cookbooks according to the precedence rules described at http://docs.opscode.com/essentials_roles.html#attribute-precedence.

See also

 

Using environments


Having separate environments for development, testing, and production is a good idea to be able to develop and test cookbook updates and other configuration changes in isolation. Chef enables you to group your nodes into separate environments to support an ordered development flow.

Getting ready

For the following examples, I assume that you have a node named my_server in the _default environment and that you have at least one cookbook (I'll use the ntp cookbook) registered with your Chef Server.

How to do it...

Let's see how to manipulate environments using Knife.

Note

This is only a good idea if you want to play around. For serious work, please create files describing your environments and put them under version control as described in the There's more... section.

  1. Create your environment on the fly using Knife. The following command will open your shell's default editor so that you can modify the environment definition:

    [email protected]:~/chef-repo $ knife environment create book
    
    {
      "name": "book",
      "description": "",
      "cookbook_versions": {
      },
      "json_class": "Chef::Environment",
      "chef_type": "environment",
      "default_attributes": {
      },
      "override_attributes": {
      }
    }
    Created book
  2. List the available environments:

    [email protected]:~/chef-repo $ knife environment list
    
    _default
    book
  3. List the nodes for all the environments:

    [email protected]:~/chef-repo $ knife node list
    my_server
  4. Verify that the node my_server is not in the book environment yet by listing nodes in the book environment only:

    [email protected]:~/chef-repo $ knife node list -E book
    
    [email protected]:~/chef-repo $
    
  5. Change the environment of the my_server node by editing the node data and changing the value of chef_environment from _default to book:

    [email protected]:~/chef-repo $ knife node edit my_server
    
    {
      "name": "my_server",
      "chef_environment": "book",
      "normal": {
      },
      "run_list": [
        "recipe[ntp]"
      ]
    }
    Saving updated chef_environment on node my_server
  6. List the nodes of the book environment again:

    [email protected]:~/chef-repo $ knife node list -E book
    my_server
  7. Use specific cookbook versions and override certain attributes for the environment:

    [email protected]:~/chef-repo $ knife environment edit book
    
    {
      "name": "book",
      "description": "",
      "cookbook_versions": {
        "ntp": "1.3.2"
      },
      "json_class": "Chef::Environment",
      "chef_type": "environment",
      "default_attributes": {
      },
      "override_attributes": {
        "ntp": {
          "servers": ["0.europe.pool.ntp.org", "1.europe.pool.ntp.org", "2.europe.pool.ntp.org", "3.europe.pool.ntp.org"]
        }
      }
    }
    Saved book

How it works...

A common use of environments is to promote cookbook updates from development to staging and then into production. Additionally, they enable you to use different cookbook versions on separate sets of nodes and also to use environment-specific attributes. You might have nodes with lesser memory in your staging environment as in your production environment. By using environment-specific default attributes, you can, for example, configure your MySQL service to consume lesser memory on staging than on production.

Note

The Chef Server always has an environment called _default which cannot be edited or deleted. All the nodes go in there if you don't specify any other environment.

Be aware that roles are not environment specific. You may use environment-specific run lists, though.

The node's environment can be queried using the node.chef_environment method inside your cookbooks.

There's more...

If you want your environments to be under version control (and you should!), a better way to create a new environment is to create a new Ruby file in the environments directory inside your Chef repository:

[email protected]:~/chef-repo $ cd environments
[email protected]:~/chef-repo $ subl book.rb
name "book"

You should add, commit, and push your new environment file to GitHub:

[email protected]:~/chef-repo $ git add environments/book.rb
[email protected]:~/chef-repo $ git commit -a -m "the book env"
[email protected]:~/chef-repo $ git push

Now, you can create the environment on the Chef Server from the newly created file using Knife:

[email protected]:~/chef-repo $ knife environment from file book.rb
Created Environment book

There is a way to migrate all the nodes from one environment to another using knife exec:

[email protected]:~/chef-repo $ knife exec -E 'nodes.transform("chef_environment:_default") { |n| n.chef_environment("book")}

You can limit your search for nodes in a specific environment:

[email protected]:~/chef-repo $ knife search node "chef_environment:book"
1 item found

See also

 

Freezing cookbooks


Uploading broken cookbooks overriding your working ones is a major pain and can result in widespread outrage throughout your infrastructure. If you've a cookbook version known to work, it is a good idea to freeze this version so that no one can overwrite the same version with broken code. When used together with environments, freezing cookbooks can keep your production servers safe.

Getting ready

Make sure you've at least one cookbook (I'll use the ntp cookbook) registered with your Chef Server.

How to do it...

Let's see what happens if we freeze a cookbook.

  1. Upload a cookbook and freeze it:

    [email protected]:~/chef-repo $ knife cookbook upload ntp --freeze
    
    Uploading ntp            [1.3.2]
    Uploaded 1 cookbook.
  2. Try to upload the same cookbook version again:

    [email protected]:~/chef-repo $ knife cookbook upload ntp
    
    Uploading ntp            [1.3.2]
    Conflict: The cookbook ntp at version 1.3.2 is frozen. Use the 'force' option to override.
  3. Change the cookbook version:

    [email protected]:~/chef-repo $ subl cookbooks/ntp/metadata.rb
    
    …
    version           "1.3.3"
  4. Upload the cookbook again:

    [email protected]:~/chef-repo $ knife cookbook upload ntp
    
    Uploading ntp            [1.3.2]
    Uploaded 1 cookbook.

How it works...

By using the --freeze option when uploading a cookbook, you tell the Chef Server that it should not accept any changes to the same version of the cookbook anymore. This is important if you're using environments and want to make sure that your production environment cannot be broken by uploading a corrupted cookbook with the same version number as used on your production servers.

By changing the version number of your cookbook, you can upload the new version. Then you can make, for example, your staging environment use that new cookbook version.

There's more...

For supporting a more elaborate workflow, you can use the knife-spork Knife plugin. It helps multiple developers work on the same Chef Server and repository without treading on each other's toes. You can find more on it at https://github.com/jonlives/knife-spork.

See also

 

Running Chef Client as a daemon


While you can run Chef Client on your nodes manually whenever you change something in your Chef repository, it's sometimes preferable to have Chef Client run automatically ever so often. Letting Chef Client run automatically makes sure that no box misses any updates.

Getting ready

You need to have a node registered with your Chef Server. It needs to be able to run chef-client without any errors.

How to do it...

Let's see how to start Chef Client in daemon mode so that it runs automatically.

  1. Start Chef Client in daemon mode, running every 30 minutes:

    [email protected]:~$ sudo chef-client -i 1800
    
  2. Validate that the Chef Client is running as a daemon:

    [email protected]:~$ ps auxw | grep chef-client
    

How it works...

The -i parameter will start Chef Client as a daemon. The given number is the seconds between each Chef Client run. In the previous example, we specified 1,800 seconds, which results in Chef Client running every 30 minutes.

You can use the same command in a service startup script.

There's more...

Instead of running Chef Client as a daemon, you can use a cron job to run it every so often:

[email protected]:~$ subl /etc/cron.d/chef_client 
PATH=/usr/local/bin:/usr/bin:/bin
# m h dom mon dow user command
*/15 * * * * root chef-client -l warn | grep -v 'retrying [1234]/5 in'

This cron job will run Chef Client every 15 minutes and swallow the first four retrying warning messages. This is important to avoid cron sending out e-mails if the Chef Server is a little slow and the Chef Client needs a few retries.

Note

It is possible to initiate a Chef Client run at any time by sending the SIGUSR1 signal to the Chef Client daemon:

[email protected]:~$ sudo killall -USR1 chef-client
 

Using the Chef console (Chef Shell)


Writing cookbooks is hard. What makes it even harder is the long feedback cycle of uploading them to the Chef Server, provisioning a Vagrant VM, checking how they failed there, rinse, and repeat. It would be so much easier if we could try out some pieces of the recipes we're writing before we've to do all this heavy lifting.

Chef comes with Chef Shell, which is essentially an interactive Ruby session with Chef. In the Chef Shell, you can create attributes, write recipes, and initialize Chef runs, among other things. It's there to evaluate parts of your recipes on the fly before you upload them to your Chef Server and execute complete cookbooks on your nodes.

How to do it...

Running the Chef Shell is straightforward.

  1. Start the Chef Shell in standalone mode:

    [email protected]:~/chef-repo $ chef-shell
    
    loading configuration: none (standalone chef-shell session)
    Session type: standalone
    Loading...[2012-12-12T20:48:01+01:00] INFO: Run List is []
    [2012-12-12T20:48:01+01:00] INFO: Run List expands to []
    done.
    
    This is chef-shell, the Chef Shell.
     Chef Version: 11.0.0
     http://www.opscode.com/chef
     http://wiki.opscode.com/display/chef/Home
    
    run `help' for help, `exit' or ^D to quit.
    
    Ohai2u [email protected]!
    chef >
    
  2. Switch to the attributes mode in the Chef Shell:

    chef > attributes_mode
    
  3. Set an attribute value to be used inside the recipe later:

    chef:attributes > set[:title] = "Chef Cookbook"
     => "Chef Cookbook" 
    chef:attributes > quit
     => :attributes 
    chef >
    
  4. Switch to the recipe mode:

    chef > recipe_mode
    
  5. Create a file resource, using the title attribute as content:

    chef:recipe > file "/tmp/book.txt" do
    chef:recipe >     content node.title
    chef:recipe ?> end
     => <file[/tmp/book.txt] @name: "/tmp/book.txt" @noop: nil @before: nil @params: {} @provider: Chef::Provider::File @allowed_actions: [:nothing, :create, :delete, :touch, :create_if_missing] @action: "create" @updated: false @updated_by_last_action: false @supports: {} @ignore_failure: false @retries: 0 @retry_delay: 2 @source_line: "(irb#1):1:in `irb_binding'" @elapsed_time: 0 @resource_name: :file @path: "/tmp/book.txt" @backup: 5 @diff: nil @cookbook_name: nil @recipe_name: nil @content: "Chef Cookbook"> 
    chef:recipe >
    
  6. Initiate a Chef run to create the file with the given content:

    chef:recipe > run_chef
    
    [2012-12-12T21:07:49+01:00] INFO: Processing file[/tmp/book.txt] action create ((irb#1) line 1)
    --- /var/folders/1r/_35fx24d0y5g08qs131c33nw0000gn/T/chef-tempfile20121212-11348-dwp1zs	2012-12-12 21:07:49.000000000 +0100
    +++ /var/folders/1r/_35fx24d0y5g08qs131c33nw0000gn/T/chef-diff20121212-11348-hdzcp1	2012-12-12 21:07:49.000000000 +0100
    @@ -0,0 +1 @@
    +Chef Cookbook
    \ No newline at end of file
    [2012-12-12T21:07:49+01:00] INFO: entered create
    [2012-12-12T21:07:49+01:00] INFO: file[/tmp/book.txt] created file /tmp/book.txt

How it works...

The Chef Shell starts an interactive Ruby (IRB) session enhanced with some Chef specific features. It offers certain modes such as attributes_mode or recipe_mode, which enable you to write commands like you would put them into an attributes file or recipe.

Entering a resource command into the recipe context will create the given resource, but not run it yet. It's like Chef reading your recipe files and creating the resources but not yet running them. You can run all the resources you created within the recipe context using the run_chef command. This will execute all the resources on your local box and physically change your system. For playing around with temporary files, your local box might do, but if you're going to do more invasive stuff such as installing or removing packages, installing services, and so on, you might want to use the Chef Shell from within a Vagrant VM.

There's more...

You can not only run the Chef Shell in standalone mode but also in Chef Solo mode and Chef Client mode. If you run it in Chef Client mode, it will load the complete run list of your node and you'll be able to tweak it inside the Chef Shell. You start the Chef Client mode by using the --client parameter:

[email protected]:~/chef-repo $ chef-shell --client

You can configure which client your Chef Shell shall act as, as well as the Chef Server to connect to in a file called chef_shell.rb.

Additionally to evaluating recipe code within your Chef Shell, you can even use it to manage your Chef Server, for example, listing all nodes:

chef > nodes.list
=> [node[my_server]]

See also

About the Author

  • Matthias Marschall

    Matthias Marschall is a Software Engineer "made in Germany". His four children make sure that he feels comfortable in lively environments, and stays in control of chaotic situations. A lean and agile engineering lead, he's passionate about continuous delivery, infrastructure automation, and all things DevOps.

    In recent years, Matthias has helped build several web-based businesses, first with Java and then with Ruby on Rails. He quickly grew into system administration, writing his own configuration management tool before migrating his whole infrastructure to Chef in its early days.

    In 2008, he started a blog (http://www.agileweboperations.com) together with Dan Ackerson. There, they have shared their ideas about DevOps since the early days of the continually emerging movement. You can find him on Twitter as @mmarschall.

    Matthias holds a Master's degree in Computer Science (Dipl.-Inf. (FH)) and teaches courses on Agile Software Development at the University of Augsburg.

    When not writing or coding, Matthias enjoys drawing cartoons and playing Go. He lives near Munich, Germany.

    Browse publications by this author

Latest Reviews

(1 reviews total)
Out of date, the book does not keep up with the Chef material.