Search icon CANCEL
Subscription
0
Cart icon
Close icon
You have no products in your basket yet
Save more on your purchases!
Savings automatically calculated. No voucher code required
Arrow left icon
All Products
Best Sellers
New Releases
Books
Videos
Audiobooks
Learning Hub
Newsletters
Free Learning
Arrow right icon
$9.99 | ALL EBOOKS & VIDEOS
Save more on purchases! Buy 2 and save 10%, Buy 3 and save 15%, Buy 5 and save 20%
Puppet 3 Cookbook - Second Edition
Puppet 3 Cookbook - Second Edition

Puppet 3 Cookbook: An essential book if you have responsibility for servers. Real-world examples and code will give you Puppet expertise, allowing more control over servers, cloud computing, and desktops. A time-saving, career-enhancing tutorial, Second Edition

By John Arundel
$28.99 $9.99
Book Aug 2013 274 pages 2nd Edition
eBook
$28.99 $9.99
Print
$48.99
Subscription
$15.99 Monthly
eBook
$28.99 $9.99
Print
$48.99
Subscription
$15.99 Monthly

What do you get with eBook?

Product feature icon Instant access to your Digital eBook purchase
Product feature icon Download this book in EPUB and PDF formats
Product feature icon Access this title in our online reader with advanced features
Product feature icon DRM FREE - Read whenever, wherever and however you want
Buy Now
Table of content icon View table of contents Preview book icon Preview Book

Puppet 3 Cookbook - Second Edition

Chapter 1. Puppet Infrastructure

 

Computers in the future may have as few as 1,000 vacuum tubes and weigh only 1.5 tons.

 
 --Popular Mechanics, 1949

In this chapter, we will cover:

  • Installing Puppet

  • Creating a manifest

  • Managing your manifests with Git

  • Creating a decentralized Puppet architecture

  • Writing a papply script

  • Running Puppet from cron

  • Deploying changes with Rake

  • Bootstrapping Puppet with Rake

  • Automatic syntax checking with Git hooks

Introduction


Some of the recipes in this book represent best practices as agreed upon by the Puppet community. Others are tips and tricks which will make it easier for you to work with Puppet, or introduce you to features that you may not have been previously aware of. Some recipes are shortcuts that I wouldn't recommend you use as standard operating procedure, but may be useful in emergencies. Finally, there are some experimental recipes that you may like to try, but are only useful or applicable in very large-scale infrastructures or otherwise unusual circumstances.

My hope is that, by reading through and thinking about the recipes presented here, you will gain a deeper and broader understanding of how Puppet works and how you can use it to improve your infrastructure. Only you can decide whether a particular recipe is appropriate for you and your organization, but I hope this collection will inspire you to experiment, find out more, and most of all have fun using Puppet!

Linux distributions

Because Linux distributions, such as Ubuntu, Red Hat, and CentOS differ in the specific details of package names, configuration file paths, and many other things, I have decided that for reasons of space and clarity the best approach for this book is to pick one distribution (Ubuntu 12.04 Precise) and stick to that. However, Puppet runs on most popular operating systems, so you should have very little trouble adapting the recipes to your own favored OS and distribution.

Puppet versions

At the time of writing, Puppet 3.2 is the latest stable version available, and consequently I have chosen that as the reference version of Puppet used in the book. The syntax of Puppet commands changes often, so be aware that while older versions of Puppet are still perfectly usable, they may not support all of the features and syntax described in this book.

Installing Puppet


If you already have a working Puppet installation, you can skip this section. If not, or if you want to upgrade or re-install Puppet, we'll go through the installation process step by step.

I'm using an Amazon EC2 cloud instance to demonstrate setting up Puppet, though you may prefer to use a physical server, a Linux workstation, or a virtual machine such as Vagrant, VMWare, or VirtualBox (with Internet access). I'll log in as the ubuntu user and use sudo to run commands that need root privileges (the default setup on Ubuntu).

Note

On EC2 Ubuntu images, the ubuntu user is already set up with the sudo permissions to run any commands as root. If you're using a different Linux distribution or you're not on EC2, you'll need to configure this yourself in the /etc/sudoers file.

Getting ready...

To prepare the machine for Puppet, we need to set its hostname.

  1. Set a suitable hostname for your server (ignore any warning from sudo):

    ubuntu@domU-12-31-39-09-51-23:~$ sudo hostname cookbook
    ubuntu@domU-12-31-39-09-51-23:~$ sudo su -c 'echo cookbook
      >/etc/hostname'
    sudo: unable to resolve host cookbook
    
  2. Log out and log back in to check the hostname is now correctly set:

    ubuntu@cookbook:~$
    
  3. Find out the local IP address of the server:

    ubuntu@cookbook:~$ ip addr show |grep eth0
        inet 10.96.247.132/23 brd 10.96.247.255 scope global eth0 
    
  4. Copy the IP address of your server (here it's 10.96.247.132) and add this to the /etc/hosts file so that it looks something like this (use your own hostname and domain):

    10.96.247.132 cookbook cookbook.example.com
    

How to do it...

Puppet packages for most Linux distributions, including Ubuntu, are available from Puppet Labs. Here's how to install the version for Ubuntu 12.04 Precise:

  1. Download the Puppet Labs repo package:

    ubuntu@cookbook:~$ wget http://apt.puppetlabs.com/puppetlabs-
      release-precise.deb
    
  2. Install the repo package:

    ubuntu@cookbook:~$ sudo dpkg -i puppetlabs-release-precise.deb
    Selecting previously unselected package puppetlabs-release.
    (Reading database ... 33975 files and directories currently 
      installed.)
    Unpacking puppetlabs-release (from puppetlabs-release-
        precise.deb)
    Setting up puppetlabs-release (1.0-5)
    
  3. Update your APT configuration:

    ubuntu@cookbook:~$ sudo apt-get update
    

    Note

    If you're not using Ubuntu 12.04 Precise, you can find out how to add the Puppet Labs repos package to your system here:

    http://docs.puppetlabs.com/guides/puppetlabs_package_repositories.html

  4. Install Puppet:

    ubuntu@cookbook:~$ sudo apt-get -y install puppet
    

    Note

    If you're on Mac, you can download and install suitable DMG images from Puppet Labs available at:

    https://downloads.puppetlabs.com/mac/

    If you're using Windows, you can download MSI packages from the Puppet Labs website available at:

    https://downloads.puppetlabs.com/windows/

  5. Run the following command to check that Puppet is properly installed:

    ubuntu@cookbook:~$ puppet --version
    3.2.2
    

If the version of Puppet you've installed is not exactly the same, it doesn't matter; you'll get whatever is the latest version made available by Puppet Labs. So long as your version is at least 3.0, you should have no trouble running the examples in this book.

If you have an older version of Puppet, you may find that some things don't work or work differently to the way you'd expect. I recommend that you upgrade to Puppet 3.x or later if at all possible.

Now that Puppet is set up, you can use it to make some configuration changes by creating a manifest. We'll see how to do this in the next section.

Creating a manifest


If you already have some Puppet code (known as a Puppet manifest), you can skip this section and go on to the next. If not, we'll see how to create and apply a simple manifest.

How to do it...

Follow these steps:

  1. First, let's create a suitable directory structure to keep the manifest code in:

    ubuntu@cookbook:~$ mkdir puppet
    ubuntu@cookbook:~$ cd puppet
    ubuntu@cookbook:~/puppet$ mkdir manifests
    
  2. Within your puppet directory, create the file manifests/site.pp with the following contents:

    import 'nodes.pp'
  3. Create the file manifests/nodes.pp with the following contents (use your machine's hostname in place of cookbook):

    node 'cookbook' {
      file { '/tmp/hello':
        content => "Hello, world\n",
      } 
    }
  4. Test your manifest with the puppet apply command. This will tell Puppet to read the manifest, compare it to the state of the machine, and make any necessary changes to that state:

    ubuntu@cookbook:~/puppet$ sudo puppet apply manifests/site.pp
    Notice: /Stage[main]//Node[cookbook]/File[/tmp/hello]/ensure:
      defined content as '{md5}a7966bf58e23583c9a5a4059383ff850'
    Notice: Finished catalog run in 0.06 seconds
    
  5. To see if Puppet did what we expected (create the file /tmp/hello with the contents Hello, world), run the following command:

    ubuntu@cookbook:~/puppet$ cat /tmp/hello
    Hello, world
    

Managing your manifests with Git


It's a great idea to put your Puppet manifests in a version control system such as Git or Subversion (I recommend Git) and give all Puppet-managed machines a checkout from your repository. This gives you several advantages:

  • You can undo changes and revert to any previous version of your manifest

  • You can experiment with new features using a branch

  • If several people need to make changes to the manifests, they can make them independently, in their own working copies, and then merge their changes later

  • You can use the git log feature to see what was changed, and when (and by whom)

Getting ready...

In this section we'll import your existing manifest files into Git. If you have created a puppet directory in the previous section, use that, otherwise use your existing manifest directory.

I'm going to use the popular GitHub service as my Git server. You don't have to do this, it's easy to run your own Git server but it does simplify things. If you already use Git and have a suitable server, feel free to use that instead.

Note

Note that GitHub currently only offers free repository hosting for public repositories (that is, everyone will be able to see and read your Puppet manifests). This isn't a good idea if your manifest contains secret data such as passwords. It's fine for playing and experimenting with the recipes in this book, but for production use, consider a private GitHub repo instead.

Here's what you need to do to prepare for importing your manifest:

  1. First, you'll need Git installed on your machine:

    ubuntu@cookbook:~/puppet$ sudo apt-get install git
    
  2. Next, you'll need a GitHub account (free for open-source projects, or you'll need to pay a small fee to create private repositories) and a repository. Follow the instructions at github.com to create and initialize your repository (from now on, just "repo" for short). Make sure you tick the box that says, Initialize this repository with a README.

  3. Authorize your SSH key for read/write access to the repo (see the GitHub site for instructions on how to do this).

How to do it...

You're now ready to add your existing manifests to the Git repo. We're going to clone the repo, and then move your manifest files into it, as follows:

  1. First, move your puppet directory to a different name:

    mv puppet puppet.import
    
  2. Clone the repo onto your machine into a directory named puppet (use your own repo URL, as shown on GitHub):

    ubuntu@cookbook:~$ git clone 
      git@github.com:bitfield/cookbook.git puppet
    Cloning into 'puppet'...
    remote: Counting objects: 3, done.
    remote: Total 3 (delta 0), reused 0 (delta 0)
    Receiving objects: 100% (3/3), done.
    
  3. Move everything from puppet.import to puppet:

    ubuntu@cookbook:~$ mv puppet.import/* puppet/
    
  4. Add and commit the new files to the repo, setting your Git identity details if necessary:

    ubuntu@cookbook:~$ cd puppet
    ubuntu@cookbook:~/puppet$ git status
    # On branch master
    # Untracked files:
    #   (use "git add <file>..." to include in what will be
      committed)
    #
    #       manifests/
    nothing added to commit but untracked files present (use "git
      add" to track)
    ubuntu@cookbook:~/puppet$ git add manifests/
    ubuntu@cookbook:~/puppet$ git config --global user.name "John
      Arundel"
    ubuntu@cookbook:~/puppet$ git config --global user.email 
      "john@bitfieldconsulting.com"
    ubuntu@cookbook:~/puppet$ git commit -m "Importing"
    [master a063a5b] Importing
    Committer: John Arundel <john@bitfieldconsulting.com>
    2 files changed, 6 insertions(+)
    create mode 100644 manifests/nodes.pp
    create mode 100644 manifests/site.pp
    
  5. Finally, push your changes back to GitHub:

    ubuntu@cookbook:~/puppet$ git push -u origin master
    Counting objects: 6, done.
    Compressing objects: 100% (4/4), done.
    Writing objects: 100% (5/5), 457 bytes, done.
    Total 5 (delta 0), reused 0 (delta 0)
    To git@github.com:bitfield/cookbook.git
       6d6aa51..a063a5b  master -> master
    

How it works...

Git tracks changes to files, and stores a complete history of all changes. The history of the repo is made up of commits. A commit represents the state of the repo at a particular point in time, which you create with the git commit command and annotate with a message.

You've added your Puppet manifest files to the repo and created your first commit. This updates the history of the repo, but only in your local working copy. To synchronize the changes with GitHub's copy, the git push command pushes all changes made since the last sync.

There's more...

Now that you have a central Git repo for your Puppet manifests, you can check out multiple copies of it in different places and work on them, before committing your changes. For example, if you're working in a team, each member can have her own local copy of the repo and synchronize changes with the others via GitHub.

Now that you've taken control of your manifests with Git, you can use it as a simple, scalable way to distribute manifest files to lots of machines. We'll see how to do this in the next section.

Creating a decentralized Puppet architecture


Some systems work best when they're decentralized. The Mafia is a good example, although, of course, there is no Mafia.

A common way to use Puppet is to run a Puppet Master server, which Puppet clients can then connect to and receive their manifests. However, you don't need a Puppet Master to use Puppet. You can run the puppet apply command directly on a manifest file to have Puppet apply it:

ubuntu@cookbook:~/puppet$ puppet apply manifests/site.pp
Notice: Finished catalog run in 0.08 seconds

In other words, if you can arrange to distribute a suitable manifest file to a client machine, you can have Puppet execute it directly without the need for a central Puppet Master. This removes the performance bottleneck of a single master server, and also eliminates a single point of failure. It also avoids having to sign and exchange the SSL certificates when provisioning a new client machine.

There are many ways you could deliver the manifest file to the client, but Git (or any version control system) does most of the work for you. You can edit your manifests in a local working copy, commit them to Git, and push them to a central repo, and from there they can be automatically distributed to the client machines.

Getting ready

If your Puppet manifests aren't already in Git, follow the steps in Managing your manifests with Git.

You'll need a second machine to check out a copy of your Puppet repo. If you're using EC2 instances, create another instance, and call it something like cookbook2.

How to do it...

Follow these steps:

  1. Check out your GitHub repo on the new machine:

    ubuntu@cookbook2:~$ git clone 
      git@github.com:bitfield/cookbook.git puppet
    Cloning into 'puppet'...
    remote: Counting objects: 8, done.
    remote: Compressing objects: 100% (5/5), done.
    remote: Total 8 (delta 0), reused 5 (delta 0)
    Receiving objects: 100% (8/8), done.
    
  2. Modify your manifests/nodes.pp file, as follows:

    node 'cookbook', 'cookbook2' {
      file { '/tmp/hello':
        content => "Hello, world\n",
      }
    }
  3. Run the following command:

    ubuntu@cookbook2:~/puppet$ sudo puppet apply manifests/site.pp
    Notice: /Stage[main]//Node[cookbook2]/File[/tmp/hello]/ensure:
    defined content as '{md5}a7966bf58e23583c9a5a4059383ff850'
    Notice: Finished catalog run in 0.05 seconds
    

How it works...

We've created a new working copy of the Puppet repo on the new machine:

ubuntu@cookbook2:~$ git clone git@github.com:bitfield/cookbook.git puppet

However, before we can run Puppet, we have to create a node declaration for the cookbook2 node:

node 'cookbook', 'cookbook2' {
  ...
}

Now, we apply the manifest:

ubuntu@cookbook2:~/puppet$ sudo puppet apply manifests/site.pp

Puppet finds the node declaration for cookbook2 and applies the same manifest that we used before on cookbook:

Notice: /Stage[main]//Node[cookbook2]/File[/tmp/hello]/ensure:
  defined content as '{md5}a7966bf58e23583c9a5a4059383ff850'

There's more...

Having scaled your Puppet infrastructure from one machine to the other, you can now extend it to as many as you like! All you need to do is check out the Git repo on a machine, and run puppet apply.

This is a great way to add Puppet management to your existing machines without lots of complicated setup, or using an extra machine to serve as a Puppet Master. Many of my clients have switched to using a Git-based infrastructure because it's simpler, easier to scale, and easier to maintain.

A refinement which you might like to consider is having each machine automatically pull changes from GitHub and apply them with Puppet. Then all you need to do is push a change to GitHub, and it will roll out to all your Puppet-managed machines within a certain time. We'll see how to do this in the following sections.

Writing a papply script


We'd like to make it as quick and easy as possible to apply Puppet on a machine, so I usually write a little script that wraps the puppet apply command with the parameters it needs. And to deploy the script where it's needed, what better tool than Puppet itself?

How to do it...

Follow these steps:

  1. In your Puppet repo, create the directories needed for a puppet module:

    ubuntu@cookbook:~/puppet$ mkdir modules
    ubuntu@cookbook:~/puppet$ mkdir modules/puppet
    ubuntu@cookbook:~/puppet$ mkdir modules/puppet/manifests
    ubuntu@cookbook:~/puppet$ mkdir modules/puppet/files 
    
  2. Create the file modules/puppet/files/papply.sh with the following contents (change the path /home/ubuntu/puppet to where your Puppet repo is located). The sudo puppet apply command should all be on one line:

    #!/bin/sh
    sudo puppet apply /home/ubuntu/puppet/manifests/site.pp
      --modulepath=/home/ubuntu/puppet/modules/ $*
  3. Create the file modules/puppet/manifests/init.pp with the following contents:

    class puppet {
      file { '/usr/local/bin/papply':
        source => 'puppet:///modules/puppet/papply.sh',
        mode   => '0755',
      }
    }
  4. Modify your manifests/nodes.pp file as follows:

    node 'cookbook' {
      include puppet
    }
  5. Apply your changes:

    ubuntu@cookbook:~/puppet$ sudo puppet apply manifests/site.pp
      --modulepath=/home/ubuntu/puppet/modules
    Notice: /Stage[main]/Puppet/File[/usr/local/bin/papply]
      /ensure: defined content as '{md5}
        171896840d39664c00909eb8cf47a53c'
    Notice: Finished catalog run in 0.07 seconds
    
  6. Test that the script works:

    ubuntu@cookbook:~/puppet$ papply
    Notice: Finished catalog run in 0.07 seconds
    

Now whenever you need to run Puppet, you can simply run papply. In future, when we apply Puppet changes, I'll ask you to run papply instead of the full puppet apply command.

How it works...

As you've seen, to run Puppet on a machine and apply a specified manifest file, we use the puppet apply command:

puppet apply manifests/site.pp

When you're using modules (such as the puppet module we just created) you also need to tell Puppet where to search for modules, using the modulepath argument:

puppet apply manifests/nodes.pp --
  modulepath=/home/ubuntu/puppet/modules

In order to run Puppet with the root privileges it needs, we have to put sudo before everything:

sudo puppet apply manifests/nodes.pp --
  modulepath=/home/ubuntu/puppet/modules

Finally, any additional arguments passed to papply will be passed through to Puppet itself, by adding the $* parameter:

sudo puppet apply manifests/nodes.pp --
  modulepath=/home/ubuntu/puppet/modules $*

That's a lot of typing, so putting this in a script makes sense. We've added a Puppet file resource that will deploy the script to /usr/local/bin and make it executable:

file { '/usr/local/bin/papply':
  source => 'puppet:///modules/puppet/papply.sh',
  mode   => '0755',
}

Finally, we include the puppet module in our node declaration for cookbook:

node 'cookbook' {
  include puppet
}

You can do the same for any other nodes managed by Puppet.

Running Puppet from cron


You can do a lot with the setup you already have: work on your Puppet manifests as a team, communicate changes via GitHub, and manually apply them on a machine using the papply script.

However, you still have to log into each machine to update the Git repo and re-run Puppet. It would be helpful to have each machine update itself and apply any changes automatically. Then all you need to do is to push a change to the repo, and it will go out to all your machines within a certain time.

The simplest way to do this is with a cron job that pulls updates from the repo at regular intervals and then runs Puppet if anything has changed.

Getting ready...

You'll need the Git repo we set up in Managing your manifests with Git and Creating a decentralized Puppet architecture, and the papply script from Writing a papply script.

You'll also need to create an SSH key that each machine can use to pull changes from the Git repo. To create this, follow these steps:

  1. Run the following command to generate the keyfile:

    ubuntu@cookbook:~/puppet$ ssh-keygen -f ubuntu
    Generating public/private rsa key pair.
    Enter passphrase (empty for no passphrase):
    Enter same passphrase again:
    Your identification has been saved in ubuntu.
    Your public key has been saved in ubuntu.pub.
    The key fingerprint is:
    ae:80:48:1c:14:51:d6:b1:73:4f:60:e2:cf:3d:ce:f1 ubuntu@cookbook
    The key's randomart image is:
    +--[ RSA 2048]----+
    | ++o.o.o         |
    |      +          |
    |     +           |
    |      = +        |
    | o     oS=       |
    |        o +      |
    |         o E     |
    |                 |
    |                 |
    +-----------------+
    
  2. Print the contents of the ubuntu.pub file:

    ubuntu@cookbook:~/puppet$ cat ubuntu.pub
    ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC8EsdLAZHIg1nnMzJuIQ5jEcFL1WI5AVhml6Z3Gw4zc4xw6F1Citomc+3DexcaD+y3VrD3WEOGcXweCsxJF0EGyJoc4RbPAJaP3D4V/+9FQVZcH90GukasvtIrfJYy2KFfRBROKtrfckMbBlWF7U2U+FwaalMOtgLzZeECSDU4eYuheN3UVcyg9Zx87zrLYU5EK1JH2WVoZd3UmdH73/rwPJWtSEQ3xs9A2wMr0lJsCF4CcFCVwrAIoEf5WzIoHbhWyZaVyPR4gHUHd3wNIzC0rmoRiYwE5uYvVBObLN10uZhn7zGPWHEc5tYU7DMbz61iTe4NLtauwJkZxmXUiPJh ubuntu@cookbook
    

Copy this and add it to your GitHub repo as a deploy key (refer to the GitHub site for instructions on how to do this). This will authorize the key to clone the Puppet repo from GitHub.

How to do it...

Follow these steps:

  1. Move the public key file into your puppet module:

    ubuntu@cookbook:~/puppet$ mv ubuntu.pub
      modules/puppet/files/ubuntu.pub
    
  2. Keep the private key file somewhere separate from your Puppet repo (you'll distribute this via some other channel to machines which need to check out the repo).

  3. Create the file modules/puppet/files/pull-updates.sh with the following contents:

    #!/bin/sh
    cd /home/ubuntu/puppet
    git pull && /usr/local/bin/papply
  4. Modify the file modules/puppet/manifests/init.pp to look like this:

    class puppet {  
      file { '/usr/local/bin/papply':
        source => 'puppet:///modules/puppet/papply.sh',
        mode   => '0755',
      }
    
      file { '/usr/local/bin/pull-updates':
        source => 'puppet:///modules/puppet/pull-updates.sh',
        mode   => '0755',
      }
      
      file { '/home/ubuntu/.ssh/id_rsa':
        source => 'puppet:///modules/puppet/ubuntu.priv',
        owner  => 'ubuntu',
        mode   => '0600',
      }
    
      cron { 'run-puppet':
        ensure  => 'present',
        user    => 'ubuntu',
        command => '/usr/local/bin/pull-updates',
        minute  => '*/10',
        hour    => '*',
      }
    }
  5. Run Puppet:

    ubuntu@cookbook:~/puppet$ papply
    Notice: /Stage[main]/Puppet/Cron[run-puppet]/ensure: created
    Notice: /Stage[main]/Puppet/File[/usr/local/bin/pull-
      updates]/ensure: defined content as 
        '{md5}20cfc6cf2a40155d4055d475a109137d'
    Notice:
      /Stage[main]/Puppet/File[/home/ubuntu/.ssh/id_rsa]/ensure:
        defined content as '{md5}db19f750104d3bf4e2603136553c6f3e'
    Notice: Finished catalog run in 0.27 seconds
    
  6. Test that the new SSH key is authorized to GitHub correctly:

    ubuntu@cookbook:~/puppet$ ssh git@github.com
    PTY allocation request failed on channel 0
    Hi bitfield/cookbook! You've successfully authenticated, but
      GitHub does not provide shell access.
    Connection to github.com closed.
    
  7. Check that the pull-updates script works properly:

    ubuntu@cookbook:~/puppet$ pull-updates
    Already up-to-date.
    Notice: Finished catalog run in 0.16 seconds
    

How it works...

Up to now, you've been using your own SSH credentials to access GitHub from the managed machine (using SSH agent forwarding), but that won't work if we want the machine to be able to pull updates unattended, while you're not logged in. So we've created a new SSH keypair and added the public part of it as a deploy key on GitHub, which gives repo access to anyone who has the private half of the key.

We've added this private key as the ubuntu user's default SSH key:

file { '/home/ubuntu/.ssh/id_rsa':
  source => 'puppet:///modules/puppet/ubuntu.priv',
  owner  => 'ubuntu',
  mode   => '0600',
}

This enables the ubuntu user to run git pull in the puppet directory. We've also added the pull-updates script, which does this and runs Puppet if any changes were pulled:

#!/bin/sh
cd /home/ubuntu/puppet
git pull && papply

We deploy this script to the box with Puppet:

file { '/usr/local/bin/pull-updates':
  source => 'puppet:///modules/puppet/pull-updates.sh',
  mode   => '0755',
}

Finally, we've created a cron job that runs pull-updates at regular intervals (every 10 minutes, but feel free to change this if you need to):

cron { 'run-puppet':
  ensure  => 'present',
  command => '/usr/local/bin/pull-updates',
  minute  => '*/10',
  hour    => '*',
}

There's more...

Congratulations, you now have a fully-automated Puppet infrastructure! Once you have checked out the repo on a new machine and applied the manifest, the machine will be set up to pull any new changes and apply them automatically.

So, for example, if you wanted to add a new user account to all your machines, all you have to do is add the account in your working copy of the manifest, and commit and push the changes to GitHub. Within 10 minutes it will automatically be applied to every machine that's running Puppet.

That's very handy, but sometimes we'd like to be able to apply the changes to a specific machine right away, without waiting for them to be picked up by the cron job. We can do this using the Rake tool, and we'll see how to do that in the next section.

Deploying changes with Rake


Rake is a useful tool from the Ruby world which you can use to help automate your Puppet workflow. Although there are lots of other ways to run commands on remote servers, this happens to be the one I use, and it's easily extensible to whatever you need it to do.

The first helpful thing we can have Rake perform for us is log into the remote machine and run the pull-updates script to apply any new Puppet manifest changes. This is fairly simple to do, as you'll see in the following sections.

Getting ready

You may already have Rake installed (try running rake), but if not, here's how to install it:

  1. Run the following command:

    sudo apt-get install rake
    

How to do it...

Perform the following steps:

  1. In your Puppet repo, create the file, Rakefile with the following contents. Replace ssh... with the correct ssh command line for you to log into your server:

    SSH = 'ssh -A -i ~/git/bitfield/bitfield.pem -l ubuntu'
    
    desc "Run Puppet on ENV['CLIENT']"
    task :apply do
      client = ENV['CLIENT']
      sh "git push"
      sh "#{SSH} #{client} pull-updates"
    end
  2. Add, commit, and push the change to the Git repo:

    ubuntu@cookbook:~/puppet$ git add Rakefile
    ubuntu@cookbook:~/puppet$ git commit -m "adding Rakefile"
    [master 63bb0c1] adding Rakefile
    1 file changed, 8 insertions(+)
    create mode 100644 Rakefile
    ubuntu@cookbook:~/puppet$ git push
    Counting objects: 31, done.
    Compressing objects: 100% (22/22), done.
    Writing objects: 100% (28/28), 4.67 KiB, done.
    Total 28 (delta 1), reused 0 (delta 0)
    To git@github.com:bitfield/cookbook.git
       a063a5b..63bb0c1  master -> master 
    
  3. On your own computer, check out a working copy of the Puppet repo, if you haven't already got one (replace the Git URL with the URL to your own repo):

    [john@Susie:~/git]$ git clone 
      git@github.com:bitfield/cookbook.git
    Cloning into 'cookbook'...
    remote: Counting objects: 36, done.
    remote: Compressing objects: 100% (26/26), done.
    remote: Total 36 (delta 1), reused 33 (delta 1)
    Receiving objects: 100% (36/36), 5.28 KiB, done.
    Resolving deltas: 100% (1/1), done.
    
  4. Run the following command, replacing cookbook with the address of your server:

    [john@Susie:~/git]$ cd cookbook
    [john@Susie:~/git/cookbook(master)]$ rake CLIENT=
      cookbook apply
    (in /Users/john/git/cookbook)
    git push
    Everything up-to-date
    ssh -A -i ~/git/bitfield/bitfield.pem -l ubuntu cookbook
      pull-updates
    Already up-to-date.
    Notice: Finished catalog run in 0.18 seconds
    Connection to cookbook closed.
    

How it works...

What you'd do manually to update your server is log into it using SSH, and then run pull-updates. The Rakefile simply automates this for you. First, we set up the correct SSH command line:

SSH = 'ssh -A -i ~/git/bitfield/bitfield.pem -l ubuntu'

The arguments to ssh are as follows:

  • -A: Forward your SSH key to the remote server, so that you can use it for further authentication

  • -i KEYFILE: Set the SSH private key to use (in this case, it's the Amazon AWS keyfile that I'm using. You may not need this argument if you're already set up for SSH access to the server with your default key.)

  • -l ubuntu: Log in as user ubuntu (the standard arrangement on EC2 servers; you may not need this argument if you log into servers using the same username as on your local machine.)

We then define a Rake task named apply:

desc "Run Puppet on ENV['CLIENT']"
task :apply do
     
end

The desc is just a helpful description, which you'll see if you run the command, rake -T, which lists available tasks:

$ rake –T
(in /Users/john/git/cookbook)
rake apply      # Run puppet on ENV['CLIENT']

The code between task and end will be run when you run rake apply. Here's what it does:

client = ENV['CLIENT']

This captures the value of the CLIENT environment variable, which tells the script the address of the remote servers to connect to.

The next line is as follows:

sh "git push"

sh simply runs a command in your local shell, in this case to make sure any changes to the Puppet repo have been pushed to GitHub. If they weren't, they wouldn't be picked up on the remote machine.

sh "#{SSH} #{client} pull-updates"

This is the line which actually connects to the client, using the ssh command line we defined at the start of the script, and the client address. Having logged in, it will then run the pull-updates command as the remote user.

Since we already set up the pull-updates script to do everything necessary to get the latest changes from GitHub and apply Puppet, that's all we need to do.

There's more...

You can now make and apply Puppet changes to your remote servers without ever explicitly logging into them. Once you've installed Puppet on a machine, checked out a copy of the manifest repo, and run Puppet for the first time, you can then do all the administration for that machine remotely.

If you're as lazy as I am, you're already asking, "Couldn't we use a Rake task to do the initial Puppet install and checkout, as well as just applying changes?"

We certainly can, and we'll see how to do that in the next section.

Bootstrapping Puppet with Rake


To make a newly provisioned machine part of our Puppet infrastructure, we just need to run a few commands on it, so let's make this process even easier by adding a new bootstrap task to the Rakefile.

Getting ready...

To get ready for the recipe, do the following:

  1. Add the following line to the top of your Rakefile:

    REPO = 'git@github.com:bitfield/cookbook.git'
  2. Add the following task anywhere in the Rakefile:

    desc "Bootstrap Puppet on ENV['CLIENT'] with
      hostname ENV['HOSTNAME']"
    task :bootstrap do
      client = ENV['CLIENT']
      hostname = ENV['HOSTNAME'] || client
      commands = <<BOOTSTRAP
    sudo hostname #{hostname} && \
    sudo su - c 'echo #{hostname} >/etc/hostname' && \
    wget http://apt.puppetlabs.com/puppetlabs-release-precise.deb && \
    sudo dpkg -i puppetlabs-release-precise.deb && \
    sudo apt-get update && sudo apt-get -y install git
      puppet && \
    git clone #{REPO} puppet && \
    sudo puppet apply --modulepath=/home/ubuntu/puppet
      /modules /home/ubuntu/puppet/manifests/site.pp
    BOOTSTRAP
      sh "#{SSH} #{client} '#{commands}'"
    end

How to do it...

You'll need a freshly provisioned server (one that you can log in to, but that doesn't have Puppet installed or any other config changes made on it). If you're using EC2, create a new EC2 instance. Get the public instance address from the AWS control panel; it'll be something like:

ec2-107-22-22-159.compute-1.amazonaws.com

Here are the steps to bootstrap the new server using Rake:

  1. Add a node declaration to your nodes.pp file for the hostname you'll be using on the new server. For example, if you wanted to call it cookbook-test, you could use

    node 'cookbook-test' {
      include puppet
    }
  2. Run the following command in the Puppet repo on your own machine (substitute the address of the new server as the value of CLIENT, and the hostname you want to use as the value of HOSTNAME). The command should all be on one line:

    $ rake CLIENT=ec2-107-22-22-159.compute-1.amazonaws.com HOSTNAME=cookbook-test bootstrap
    
  3. You'll see output something like the following:

    (in /Users/john/git/cookbook)
    ssh -A -i ~/git/bitfield/bitfield.pem -l ubuntu ec2-107-22-22-159.compute-1.amazonaws.com 'sudo hostname cookbook-test && sudo su -c 'echo cookbook-test >/etc/hostname' && wget http://apt.puppetlabs.com/puppetlabs-release-precise.deb && sudo dpkg -i puppetlabs-release-precise.deb && sudo apt-get update && sudo apt-get -y install git puppet && git clone git@github.com:bitfield/cookbook.git puppet && sudo puppet apply --modulepath=/home/ubuntu/puppet/modules /home/ubuntu/puppet/manifests/site.pp'
    The authenticity of host 'ec2-107-22-22-159.compute-1.amazonaws.com (107.22.22.159)' can't be established.
    RSA key fingerprint is 23:c5:06:ad:58:f3:8d:e5:75:bd:94:6e:1e:a0:a3:a4.
    Are you sure you want to continue connecting (yes/no)? yes
    Warning: Permanently added 'ec2-107-22-22-159.compute-1.amazonaws.com,107.22.22.159' (RSA) to the list of known hosts.
    sudo: unable to resolve host cookbook-test
    --2013-03-15 15:53:44--  http://apt.puppetlabs.com/puppetlabs-release-precise.deb
    Resolving apt.puppetlabs.com (apt.puppetlabs.com)... 96.126.116.126, 2600:3c00::f03c:91ff:fe93:711a
    Connecting to apt.puppetlabs.com (apt.puppetlabs.com)|96.126.116.126|:80... connected.
    HTTP request sent, awaiting response... 200 OK
    Length: 3392 (3.3K) [application/x-debian-package]
    Saving to: `puppetlabs-release-precise.deb'
         0K                                                       100%  302M=0s
    2013-03-15 15:53:44 (302 MB/s) - `puppetlabs-release-precise.deb' saved [3392/3392]
    Selecting previously unselected package puppetlabs-release.
    (Reading database ... 25370 files and directories currently installed.)
    Unpacking puppetlabs-release (from puppetlabs-release-precise.deb) ...
    Setting up puppetlabs-release (1.0-5) ...
    Processing triggers for initramfs-tools ...
    update-initramfs: Generating /boot/initrd.img-3.2.0-29-virtual
    Ign http://us-east-1.ec2.archive.ubuntu.com precise InRelease
    [ ... apt output redacted ... ]
    Setting up hiera (1.1.2-1puppetlabs1) ...
    Setting up puppet-common (3.2.2-1puppetlabs1) ...
    Setting up puppet (3.2.2-1puppetlabs1) ...
    * Starting puppet agent
    puppet not configured to start, please edit /etc/default/puppet to enable
       ...done.
    Processing triggers for libc-bin ...
    ldconfig deferred processing now taking place
    Cloning into 'puppet'...
    Warning: Permanently added 'github.com,207.97.227.239' (RSA) to the list of known hosts.
    Notice: /Stage[main]/Puppet/Cron[run-puppet]/ensure: created
    Notice: /Stage[main]/Puppet/File[/usr/local/bin/pull-updates]/ensure: defined content as '{md5}20cfc6cf2a40155d4055d475a109137d'
    Notice: /Stage[main]/Puppet/File[/usr/local/bin/papply]/ensure: defined content as '{md5}171896840d39664c00909eb8cf47a53c'
    Notice: /Stage[main]/Puppet/File[/home/ubuntu/.ssh/id_rsa]/ensure: defined content as '{md5}db19f750104d3bf4e2603136553c6f3e'
    Notice: Finished catalog run in 0.11 seconds
    

How it works...

Here's a line by line breakdown of what the Rake task does. In order to make the machine ready to run Puppet, we need to set its hostname to the name you've chosen:

sudo hostname #{hostname}
sudo echo #{hostname} >/etc/hostname

Next, we download and install the Puppet Labs repo package, and install Puppet and Git:

wget http://apt.puppetlabs.com/puppetlabs-release-precise.deb
sudo dpkg -i puppetlabs-release-precise.deb
sudo apt-get update && sudo apt-get -y install git puppet

We need to disable the SSH StrictHostKeyChecking option to avoid being prompted when the script clones the Git repo:

echo -e \"Host github.com\n\tStrictHostKeyChecking no\n\" 
  >> ~/.ssh/config

We check out the repo:

git clone #{REPO} puppet

And finally, run Puppet:

sudo puppet apply --modulepath=/home/ubuntu/puppet/modules
  /home/ubuntu/puppet/manifests/site.pp

The new machine will now pull and apply Puppet changes automatically, without you ever having to log into it interactively. You can use this Rake task to bring lots of new servers under Puppet control quickly.

Automatic syntax checking with Git hooks


It would be nice if we knew there was a syntax error in the manifest before we even committed it. You can have Puppet check the manifest using the puppet parser validate command:

ubuntu@cookbook:~/puppet$ puppet parser validate manifests/nodes.pp
Error: Could not parse for environment production: Syntax error at end of file; expected '}' at /home/ubuntu/puppet/manifests/nodes.pp:3
Error: Try 'puppet help parser validate' for usage

This is especially useful because a mistake anywhere in the manifest will stop Puppet from running on any node, even on nodes that don't use that particular part of the manifest. So checking in a bad manifest can cause Puppet to stop applying updates to production for some time, until the problem is discovered, and this could potentially have serious consequences. The best way to avoid this is to automate the syntax check, by using a pre-commit hook in your version control repo.

How to do it…

Follow these steps:

  1. In your Puppet repo, create a new hooks directory:

    ubuntu@cookbook:~/puppet$ mkdir hooks
    
  2. Create the file hooks/check_syntax.sh with the following contents (based on a script by Puppet Labs):

    #!/bin/sh
    
    syntax_errors=0
    error_msg=$(mktemp /tmp/error_msg.XXXXXX)
    
    if git rev-parse --quiet --verify HEAD > /dev/null
    then
        against=HEAD
    else
        # Initial commit: diff against an empty tree object
        against=4b825dc642cb6eb9a060e54bf8d69288fbee4904
    fi
    
    # Get list of new/modified manifest and template files
      to check (in git index)
    for indexfile in `git diff-index --diff-filter=AM --
      name-only --cached $against | egrep '\.(pp|erb)'`
    do
        # Don't check empty files
        if [ `git cat-file -s :0:$indexfile` -gt 0 ]
        then
            case $indexfile in
                *.pp )
                    # Check puppet manifest syntax
                    git cat-file blob :0:$indexfile | 
                      puppet parser validate > $error_msg ;;
                *.erb )
                    # Check ERB template syntax
                    git cat-file blob :0:$indexfile | 
                      erb -x -T - | ruby -c 2> $error_msg >
                        /dev/null ;;
            esac
            if [ "$?" -ne 0 ]
            then
                echo -n "$indexfile: "
                cat $error_msg
                syntax_errors=`expr $syntax_errors + 1`
            fi
        fi
    done
    
    rm -f $error_msg
    
    if [ "$syntax_errors" -ne 0 ]
    then
        echo "Error: $syntax_errors syntax errors found,
          aborting commit."
        exit 1
    fi
  3. Set execute permission for the hook script with the following command:

    ubuntu@cookbook:~/puppet$ chmod a+x .hooks/check_syntax.sh
    
  4. Add the following task to your Rakefile:

    desc "Add syntax check hook to your git repo"
    task :add_check do
      here = File.dirname(__FILE__)
      sh "ln -s #{here}/hooks/check_syntax.sh
        #{here}/.git/hooks/pre-commit"
      puts "Puppet syntax check hook added"
    end
  5. Run the following command:

    ubuntu@cookbook:~/puppet$ rake add_check
    ln -s /home/ubuntu/puppet/hooks/check_syntax.sh
      /home/ubuntu/puppet/.git/hooks/pre-commit
    Puppet syntax check hook added
    

How it works…

The check_syntax.sh script will prevent you from committing any files with syntax errors:

ubuntu@cookbook:~/puppet$ git commit -m "test commit"
Error: Could not parse for environment production: Syntax error at
  '}' at line 3
Error: Try 'puppet help parser validate' for usage
manifests/nodes.pp: Error: 1 syntax errors found, aborting commit.

If you add the hooks directory to your Git repo, anyone who has a checkout can run the rake add_check task and get this syntax checking behavior.

Left arrow icon Right arrow icon

Key benefits

  • Use Puppet 3 to take control of your servers and desktops, with detailed step-by-step instructions
  • Covers all the popular tools and frameworks used with Puppet: Dashboard, Foreman, and more
  • Teaches you how to extend Puppet with custom functions, types, and providers
  • Packed with tips and inspiring ideas for using Puppet to automate server builds, deployments, and workflows
  •  

Description

A revolution is happening in web operations. Configuration management tools can build servers in seconds, and automate your entire network. Tools like Puppet are essential to taking full advantage of the power of cloud computing, and building reliable, scalable, secure, high-performance systems. More and more systems administration and IT jobs require some knowledge of configuration management, and specifically Puppet."Puppet 3 Cookbook" takes you beyond the basics to explore the full power of Puppet, showing you in detail how to tackle a variety of real-world problems and applications. At every step it shows you exactly what commands you need to type, and includes full code samples for every recipe.The book takes the reader from a basic knowledge of Puppet to a complete and expert understanding of Puppet's latest and most advanced features, community best practices, writing great manifests, scaling and performance, and extending Puppet by adding your own providers and resources. It starts with guidance on how to set up and expand your Puppet infrastructure, then progresses through detailed information on the language and features, external tools, reporting, monitoring, and troubleshooting, and concludes with many specific recipes for managing popular applications.The book includes real examples from production systems and techniques that are in use in some of the world's largest Puppet installations, including a distributed Puppet architecture based on the Git version control system. You'll be introduced to powerful tools that work with Puppet such as Hiera. The book also explains managing Ruby applications and MySQL databases, building web servers, load balancers, high-availability systems with Heartbeat, and many other state-of-the-art techniques

What you will learn

Installing and setting up Puppet for the first time Producing eye-catching reports and information for management Understanding common error messages and troubleshooting common problems Managing large networks Taking control of configuration data with Hiera and encrypting secrets with GnuPG Producing reliable, clean, maintainable code to community standards with puppet-lint and rspec-puppet Using classes and inheritance to write powerful Puppet code Deploying configuration files and templates for lightning-fast installations Using virtual machines to build test and staging environments, and production systems on cloud platforms such as EC2 Automating every aspect of your systems including provisioning, deployment and change management Making Puppet reliable, performant, and scalable

Product Details

Country selected

Publication date : Aug 26, 2013
Length 274 pages
Edition : 2nd Edition
Language : English
ISBN-13 : 9781782169765
Vendor :
Puppet

What do you get with eBook?

Product feature icon Instant access to your Digital eBook purchase
Product feature icon Download this book in EPUB and PDF formats
Product feature icon Access this title in our online reader with advanced features
Product feature icon DRM FREE - Read whenever, wherever and however you want
Buy Now

Product Details


Publication date : Aug 26, 2013
Length 274 pages
Edition : 2nd Edition
Language : English
ISBN-13 : 9781782169765
Vendor :
Puppet

Table of Contents

16 Chapters
Puppet 3 Cookbook Chevron down icon Chevron up icon
Credits Chevron down icon Chevron up icon
About the Author Chevron down icon Chevron up icon
About the Reviewers Chevron down icon Chevron up icon
www.PacktPub.com Chevron down icon Chevron up icon
Preface Chevron down icon Chevron up icon
1. Puppet Infrastructure Chevron down icon Chevron up icon
2. Puppet Language and Style Chevron down icon Chevron up icon
3. Writing Better Manifests Chevron down icon Chevron up icon
4. Working with Files and Packages Chevron down icon Chevron up icon
5. Users and Virtual Resources Chevron down icon Chevron up icon
6. Applications Chevron down icon Chevron up icon
7. Servers and Cloud Infrastructure Chevron down icon Chevron up icon
8. External Tools and the Puppet Ecosystem Chevron down icon Chevron up icon
9. Monitoring, Reporting, and Troubleshooting Chevron down icon Chevron up icon
Index Chevron down icon Chevron up icon

Customer reviews

Top Reviews
Rating distribution
Empty star icon Empty star icon Empty star icon Empty star icon Empty star icon 0
(0 Ratings)
5 star 0%
4 star 0%
3 star 0%
2 star 0%
1 star 0%
Filter icon Filter
Top Reviews

Filter reviews by


No reviews found
Get free access to Packt library with over 7500+ books and video courses for 7 days!
Start Free Trial

FAQs

How do I buy and download an eBook? Chevron down icon Chevron up icon

Where there is an eBook version of a title available, you can buy it from the book details for that title. Add either the standalone eBook or the eBook and print book bundle to your shopping cart. Your eBook will show in your cart as a product on its own. After completing checkout and payment in the normal way, you will receive your receipt on the screen containing a link to a personalised PDF download file. This link will remain active for 30 days. You can download backup copies of the file by logging in to your account at any time.

If you already have Adobe reader installed, then clicking on the link will download and open the PDF file directly. If you don't, then save the PDF file on your machine and download the Reader to view it.

Please Note: Packt eBooks are non-returnable and non-refundable.

Packt eBook and Licensing When you buy an eBook from Packt Publishing, completing your purchase means you accept the terms of our licence agreement. Please read the full text of the agreement. In it we have tried to balance the need for the ebook to be usable for you the reader with our needs to protect the rights of us as Publishers and of our authors. In summary, the agreement says:

  • You may make copies of your eBook for your own use onto any machine
  • You may not pass copies of the eBook on to anyone else
How can I make a purchase on your website? Chevron down icon Chevron up icon

If you want to purchase a video course, eBook or Bundle (Print+eBook) please follow below steps:

  1. Register on our website using your email address and the password.
  2. Search for the title by name or ISBN using the search option.
  3. Select the title you want to purchase.
  4. Choose the format you wish to purchase the title in; if you order the Print Book, you get a free eBook copy of the same title. 
  5. Proceed with the checkout process (payment to be made using Credit Card, Debit Cart, or PayPal)
Where can I access support around an eBook? Chevron down icon Chevron up icon
  • If you experience a problem with using or installing Adobe Reader, the contact Adobe directly.
  • To view the errata for the book, see www.packtpub.com/support and view the pages for the title you have.
  • To view your account details or to download a new copy of the book go to www.packtpub.com/account
  • To contact us directly if a problem is not resolved, use www.packtpub.com/contact-us
What eBook formats do Packt support? Chevron down icon Chevron up icon

Our eBooks are currently available in a variety of formats such as PDF and ePubs. In the future, this may well change with trends and development in technology, but please note that our PDFs are not Adobe eBook Reader format, which has greater restrictions on security.

You will need to use Adobe Reader v9 or later in order to read Packt's PDF eBooks.

What are the benefits of eBooks? Chevron down icon Chevron up icon
  • You can get the information you need immediately
  • You can easily take them with you on a laptop
  • You can download them an unlimited number of times
  • You can print them out
  • They are copy-paste enabled
  • They are searchable
  • There is no password protection
  • They are lower price than print
  • They save resources and space
What is an eBook? Chevron down icon Chevron up icon

Packt eBooks are a complete electronic version of the print edition, available in PDF and ePub formats. Every piece of content down to the page numbering is the same. Because we save the costs of printing and shipping the book to you, we are able to offer eBooks at a lower cost than print editions.

When you have purchased an eBook, simply login to your account and click on the link in Your Download Area. We recommend you saving the file to your hard drive before opening it.

For optimal viewing of our eBooks, we recommend you download and install the free Adobe Reader version 9.