Chef Essentials

Chef-solo is designed for individuals who do not need a hosted installation for a large-scale infrastructure management. Typical use cases of chef-solo include developers managing virtual machines, test installations, or small-scale infrastructure management. The installation of chef-solo is as simple as installing a single Ruby gem.

For those who are not intimately familiar with Ruby, a Ruby gem is a mechanism to package, deliver, and manage Ruby code. These packages may be libraries that provide functionality for developers, or they may be composed only of scripts and tools. Chef-solo is, like many things in life, somewhere in the middle. The gem contains a set of libraries that make up the core functionality as well as a suite of scripts that are used by end users. Before you install Chef, you should consider installing Ruby Version Manager (RVM), rbenv, chruby, or another Ruby manager of your choice to keep your gem collections isolated.

A great tool to manage your gems is RVM. The simplest way to install RVM is to use the installation script provided by the development team on the RVM website (http://rvm.io). The following command will download the script and pipe it through bash:

curl -sSL https://get.rvm.io | bash -s stable

Once it is installed, you will initially need to include RVM's functionality in your shell:

source ~/.rvm/scripts/rvm

Additionally, you might need to add the previous command line to your shell's startup scripts (such as ~/.bashrc or ~/.zshrc, depending on which shell you use). Once RVM is installed, you will want to install a recent version of Ruby, for example, Ruby 1.9.3:

rvm install 1.9.3

Once Ruby 1.9 is installed, you will want to create a gemset. A gemset is RVM's way of isolating gems inside a container, and it will provide you with a place to install gems in such a way that they will not conflict with other gems. This has the benefit of allowing you to install anything you want, without requiring administrative privileges and keeping gems from conflicting with each other. A gemset can be created using the following command:

rvm use 1.9.3@chef --create

The previous command will simultaneously create the gemset named chef (if it does not exist) for your installation of Ruby 1.9.3 and then set it as the active gemset. Once you start using this new gemset, you will want to install the Chef gem—this contains chef-solo and all the command-line tools you need to work with Chef—using the gem command-line tool:

gem install chef

Now that the Chef gem is installed, it is time to verify that everything is working fine. In order to use chef-solo, you need to give the following information to it:

For simplicity, we will store all of these files inside of the chef directory in your home directory. You are free to put things where you see fit as you become more comfortable working with Chef.

In order to exercise our new tool, we will do something simple: we'll write a recipe that will create an example.txt file in your home directory. The recipe we create will be called create_file, and we'll put that recipe inside a cookbook, which will be named demo.

First, create the directory that will contain the demo cookbook's recipes (and any in between):

user@host:~ $ mkdir -p ~/chef/cookbooks/demo/recipes 

Next, add the following code to a file, create_file.rb, located in the demo cookbook directory you created at ~/chef/cookbooks/demo/recipes:

file "#{ENV['HOME']}/example.txt" do
  action :create
  content "Greetings #{ENV['USER']}!"
end

This tells Chef that we want to create a file, $HOME/example.txt. Its contents should be Greetings $USER, where $USER will be replaced with the value of $USER, typically the login name of whoever is executing the recipe.

Now we will need to create a JSON document that describes what chef-solo should execute. JSON is an acronym for JavaScript Object Notation, and Chef uses JSON extensively because it is easy to parse, human readable, and easy to generate from all sorts of tools and languages. Create a file, node.json, located in our work directory (~/chef/ in this case) and add the following content in order to tell Chef that we want to execute the newly created create_file recipe in the demo cookbook:

{   
   "run_list": [ 
       "recipe[demo::create_file]" 
   ] 
} 

Here, we are defining the node as having a run list, which is just an array of things to do, and that the run list contains one recipe, create_file, which it can find in the demo cookbook (the general form of a recipe being cookbook::recipe).

Finally, we'll tell Chef where to find the files we just created using a solo.rb file that we will store in our working directory (~/chef in our case):

CHEF_ROOT="#{ENV['HOME']}/chef"
file_cache_path "#{CHEF_ROOT}"
cookbook_path "#{CHEF_ROOT}/cookbooks"
json_attribs "#{CHEF_ROOT}/node.json" 

Now that you have populated the required configuration files, you can run chef-solo and execute the run list specified. In our case, the run list is defined as only one recipe, create_file, but can be as simple or as complex as needed. The previous configuration tells Chef to load the node configuration from the file node.json to look for cookbooks in ~/chef/cookbooks/ and to store any state data in ~/chef/. In order to execute these commands, you will want to run chef-solo:

chef-solo -c ~/chef/solo.rb 

The -c option tells chef-solo which script contains the configuration. Once you do this, you will see the actions that your recipe is performing:

Starting Chef Client, version 11.8.2
Compiling Cookbooks...
Converging 1 resources
Recipe: demo::create_file
  * file[/home/user/example.txt] action create
    - create new file /home/user/example.txt
    - update content in file /home/user/example.txt from none to b4a3cc
        --- /home/user/example.txt	2014-01-20 23:59:54.692819000 -0500
        +++ /tmp/.example.txt20140122-13411-1vxtg7v	2014-01-20 23:59:54.692819000 -0500
        @@ -1 +1,2 @@
        +Greetings user!

Chef Client finished, 1 resources updated

Once it is completed, you will see that ~/example.txt contains the greeting that you defined in the recipe. Now that you have successfully used chef-solo, let's move on to the Chef service.

Since the Chef service is designed to be a multiuser platform and provides functionalities that chef-solo does not offer, the installation is more complex and involves more software to achieve this functionality. Services such as Solr for full-text indexing and PostgreSQL for data storage can be a significant resource for consumers, so you will want to install Chef on a host with sufficient memory and disk space. A system with 2 GB of memory and 5-10 GB of disk space available will be plenty for a small to medium sized installation. You will need more resources as your requirements for data storage and indexing increase over time, so plan accordingly.

Additionally, for those who have installed the Chef server before, the installation path has been greatly simplified. In addition to replacing CouchDB with PostgreSQL as the primary data storage engine, there is now a single omnibus installation package for Chef that installs all of the requirements for Chef at a single location so that it operates in isolation and does not require dependencies to be installed separately.

In order to install the Chef service, you will need to have the following:

At the end of this section, you will have a fully functional Chef service installed and ready to work with. Before you get started, let's look at what you will be installing on your system so that you know what to expect. The components that make up a Chef service are as follows:

The Chef API service is responsible for delivering run lists and receiving information from nodes as well as providing a way for a system administrator to configure recipes, run lists, data bags, and the like. In order to generate this data, the API service relies on its persistent data storage engine, in this case PostgreSQL, to store its data. The option to search for data is provided by the Solr search engine, and RabbitMQ is responsible for gluing them all together. Together, these components provide Chef with the ability to distribute, store, index, and manage your infrastructure's configuration data.

The easiest way to install Chef is through a single downloadable package, which is provided for Ubuntu 10.04 through 12.10 and Red Hat Enterprise Linux Versions 5 and 6. This package, referred to as the omnibus installer, contains everything you need to get a server up and running. You can find it on http://www.getchef.com/chef/install/.

At the time of writing this, 11.0.10 is the latest version and is the one that will be used for this book. The newer version of the 11.x series of Chef should have a very similar, if not identical, configuration. Note that these installers are somewhat large, being that they contain all of the dependencies needed. For example, the Ubuntu 12.10 package for Chef 11.0.10 is approximately 200 MB in size.

Installation on all supported platforms is relatively similar. The only key differences are the names of the package files that you will download and the commands you will use to install Chef.

The high-level steps you will take are as follows:

Because steps 3 and 4 will be the same for both Ubuntu and Red Hat installation procedures, the instructions will be in a section following the Red Hat installation guide.

The following are instructions for an Ubuntu-based system; they were performed on an Ubuntu 12.04 host, but should be identical for all supported Ubuntu distributions. For Red Hat-based installation instructions, see the next section.

user@ubuntu:~ $ curl -O https://opscode-omnibus-packages.s3.amazonaws.com/ubuntu/12.04/x86_64/chef-server_11.0.10-1.ubuntu.12.04_amd64.deb

user@ubuntu:~ $ sudo dpkg -i chef-server_11.0.10-1.ubuntu.12.04_amd64.deb

Installation on a Red Hat Enterprise Linux distribution is as straightforward as installing any other package. You download the package to the local disk and install it using RPM tools.

user@rhel:~ $ curl -O https://opscode-omnibus-packages.s3.amazonaws.com/el/6/x86_64/chef-server-11.0.10-1.el6.x86_64.rpm

user@rhel:~ $ sudo rpm -ivh chef-server-11.0.10-1.el6.x86_64.rpm

Earlier in this chapter, you were briefly introduced to the chef-solo tool. You saw how it can be used to manage your local server using on-disk recipes and configuration data. The Chef team has leveraged this ability to do just that with the Chef server using chef-solo to bootstrap the server configuration. If you were to look at the code for the /opt/chef-server/bin/chef-server-ctl script, you would see that the last line in the script executes the following command:

/opt/chef-server/embedded/bin/omnibus-ctl chef-server /opt/chef-server/embedded/service/omnibus-ctl $@

If you follow the trail and dig into the omnibus-ctl script, you will find that it is just a wrapper around the omnibus-ctl Ruby gem. Digging into the omnibus-ctl gem, you will see that in the end, the reconfigure command you pass on the command line is a Ruby method that makes the following call:

run_command("chef-solo -c #{base_path}/embedded/cookbooks/solo.rb -j #{base_path}/embedded/cookbooks/dna.json")

This tells us that the Chef omnibus package uses chef-solo to configure itself—a pretty clever trick indeed! You can see just how powerful a tool chef-solo can be, being able to configure and reconfigure the Chef service.

What you will probably notice right away is that a lot of text is being scrolled past in your terminal window. If you were to look at the contents, you would see that it shows you the actions that are being taken by chef-solo to provision your new services. As there is a lot of information going past (thousands of lines), here is a high-level overview of what is happening on your host:

Clearly, there is a lot happening here; if you have any outstanding concerns about what is being done, be sure to read through the output. One of the great things about Chef is that the recipes are just a set of scripts that you can open and view the contents of, and the output shows you what is happening during the execution. Everything it does is transparent and manageable by you.

Once the configuration of your services is complete, you will want to validate that the required services are running. Again, the chef-server-ctl script will be used, but we will invoke the status subcommand instead of the reconfigure subcommand, as shown in the following code:

user@host:~ $ sudo chef-server-ctl status
run: bookshelf: (pid 3901) 3123s; run: log: (pid 3900) 3123s
run: chef-expander: (pid 3861) 3129s; run: log: (pid 3860) 3129s
run: chef-server-webui: (pid 4053) 3095s; run: log: (pid 4052) 3095s
run: chef-solr: (pid 3819) 3135s; run: log: (pid 3818) 3135s
run: erchef: (pid 4230) 3062s; run: log: (pid 3937) 3117s
run: nginx: (pid 4214) 3064s; run: log: (pid 4213) 3064s
run: postgresql: (pid 3729) 3146s; run: log: (pid 3728) 3146s
run: rabbitmq: (pid 3423) 3172s; run: log: (pid 3422) 3172s

The status subcommand will show you the process ID of each component, how long it has been running for, the PID of the logging process associated with that service, and how long the logging service has been running. For example, we can see that chef-server-webui has a PID of 4053 and has been running for close to an hour, and the logger has a PID of 4052, having been running for just as long as the service.

As you can see, the installation of Chef yields a number of components that will need to be up and running in order to successfully use Chef. You should have the following components running and listening on the following network ports:

Public components need to be made available to any clients, nodes, or end users that expect to use the Chef service over the network. Configuring your infrastructure to ensure that your services are available via the network is outside of the scope of this book as there are a near-infinite number of possible network configurations.

At a higher level, make sure that any firewall devices or packet-filtering systems are not preventing traffic from reaching these services if you see that they are running, but are having difficulties in connecting to them. If any of these services are not running, you will need to consult the logfiles generated by the service to determine what might be preventing them from starting up.

After setting up knife, we can use it to validate that it was configured correctly by querying the Chef server using some simple commands. The knife commands follow the format knife <command> <subcommand>, where command is either a client, configuration, cookbook, cookbook site, data bag, environment, exec, help, index, node, recipe, role, search, ssh, status, or tag. Subcommands will vary with the command, but they typically include things such as show, create, list, and delete (among others).

As there will initially be no nodes, cookbooks, recipes, roles, data bags, and such, we will query the list of clients that the server knows about. This should be a list of two clients: chef-webui (as it is a consumer of the API itself) and chef-validator (without it, it wouldn't be possible to register a new client).

The client command, with the list subcommand, prints a list of clients that the server knows about. At this point, running the command would look like this:

user@chef:~$ knife client list
chef-validator
chef-webui
user@chef:~$

Once you know that it works, you can use knife to interact with the API. Unfortunately, we do not have much data in the system just yet, but we can use the show subcommand in conjunction with the client command and a client name to display more detailed information about a client:

user@chef:~$ knife client show chef-webui
admin:      true
chef_type:  client
json_class: Chef::ApiClient
name:       chef-webui
public_key: -----BEGIN PUBLIC KEY-----
MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAos5cQ1NxP7zKf1zRM33g
YeVyHNOO5NcICjSIvqQ5A37wwLfgtPLJQqboW7ZcNL3xYcKOlfYSEK7xha3ss8tT
A+XMifaFp3JsdheyPeIJir2bc9iltUUcbpw9PJ2aQKTBlFNx23A7ag+zBfxcDjbY
7RkdcziwB74ynd6e/K8c0JTRnA5NxoHkFc6v8a/itwujGwugWJXDQunWfCmAvjws
JgDOUu2aHOCVIVkc8it51Sc7Anx0YnCjNmdhz1xIo0MOVNOEmC9ypP0Z7mVv1C69
WWBOEvS9zimjXo4rxBwFmWkPEIG6yPQjhuNmFd69K14vZQtAsH07AZFRSS7HLWnZ
WQIDAQAB
-----END PUBLIC KEY-----

validator:  false
user@chef:~$