Puppet: Integrating External Tools

Share this page via Facebook, Twitter or LinkedIn.

Your message has been sent.

How would you like to send this article:

Save this article to your account for easy access.

by John Arundel | November 2011 | Cookbooks Open Source

Puppet is a useful tool by itself, but you can get much greater benefits from using Puppet in combination with other tools and frameworks. We'll take a look at some of these.

In this article by John Arundel, author of Puppet 2.7 Cookbook,we will cover the following topics:

Executing commands before and after Puppet runs
Using Public modules
Creating your own resource types
Using MCollective

Executing commands before and after Puppet runs

If you need to have a command executed before each Puppet run, you can do this using the prerun_command configuration setting. Similarly, you can use postrun_command to execute a command after the run has completed. This mechanism gives you a powerful hook to integrate Puppet with other software, or even trigger events on other machines. The prerun and postrun commands must succeed (that is, return a zero exit status), or Puppet will report an error. This enables you to have any command failures reported using Puppet's reporting mechanism, for example.

How to do it...

Set prerun_command or postrun_command in puppet.conf to the commands you want to run:

prerun_command = /usr/local/bin/before-puppet-run.sh postrun_command = /usr/local/bin/after-puppet-run.sh

There's more

You can use prerun and postrun commands to integrate Puppet with Ubuntu's etckeeper system. Etckeeper is a version control system for tracking changes to files in the /etc directory. To do this, define these commands in puppet.conf:

prerun_command=/etc/puppet/etckeeper-commit-pre postrun_command=/etc/puppet/etckeeper-commit-post

Using public modules

"Plagiarize, plagiarize, plagiarize / Only be sure always to call it please 'research' "—Tom Lehrer, 'Lobachevsky'

If in doubt, steal. In many cases when you write a Puppet module to manage some software or service, you don't have to start from scratch. Community-contributed modules are available at the Puppet Forge site for many popular applications. Sometimes, a community module will be exactly what you need and you can download and start using it straight away. In other cases, you will need to make some modifications to suit your particular needs and environment. If you are new to Puppet, it can be a great help to have some existing code to start with. On the other hand, community modules are often written to be as general and portable as possible, and the extra code required can make them harder to understand. In general I would not recommend treating Puppet Forge as a source of 'drop-in' modules which you can deploy without reading or understanding the code. This introduces an external dependency to your Puppet infrastructure, and doesn't help advance your understanding and experience of Puppet. Rather, I would use it as a source of inspiration, help, and examples. A module taken from Puppet Forge should be a jumping-off point for you to develop and improve your own modules. Be aware that a given module may not work on your Linux distribution. Check the README file which comes with the module to see if your operating system is supported.

Getting ready

The easiest way to use Puppet Forge modules is to install the puppet-module tool:

# gem install puppet-module Fetching: puppet-module-0.3.2.gem (100%) ****************************************************************************** Thank you for installing puppet-module from Puppet Labs! * Usage instructions: read "README.markdown" or run `puppet-module usage` * Changelog: read "CHANGES.markdown" or run `puppet-module changelog` * Puppet Forge: visit http://forge.puppetlabs.com/ ****************************************************************************** Successfully installed puppet-module-0.3.2 1 gem installed Installing ri documentation for puppet-module-0.3.2... Installing RDoc documentation for puppet-module-0.3.2...

Run puppet-module to see the available commands:

# puppet-module Tasks: puppet-module build [PATH_TO_MODULE] # Build a module for release puppet-module changelog # Display the changelog for this tool puppet-module changes [PATH_TO_MODULE] # Show modified files in an installed m... puppet-module clean # Clears module cache for all repositories puppet-module generate USERNAME-MODNAME # Generate boilerplate for a new module puppet-module help [TASK] # Describe available tasks or one speci... puppet-module install MODULE_NAME_OR_FILE [OPTIONS] # Install a module (eg, 'user-modname')... puppet-module repository # Show currently configured repository puppet-module search TERM # Search the module repository for a mo... puppet-module usage # Display detailed usage documentation ... puppet-module version # Show the version information for this... Options: -c, [--config=CONFIG] # Configuration file # Default: /etc/puppet/puppet.conf

How to do it

In this example, we'll use puppet-module to find and install a module to manage the Tomcat application server.

Search for a suitable module as follows:

# puppet-module search tomcat
=====================================
Searching http://forge.puppetlabs.com
-------------------------------------
2 found.
--------
camptocamp/tomcat (0.0.1)
jeffmccune/tomcat (1.0.1)

In this example we'll install the Jeff McCune version:

# cd /etc/puppet/modules # puppet-module install jeffmccune/tomcat Installed "jeffmccune-tomcat-1.0.1" into directory: jeffmccune-tomcat

The module is now ready to use in your manifests: looking at the source code will show you how to do this.

How it works...

The puppet-module tool simply automates the process of searching and downloading modules from the Puppet Forge site. You can browse the site to see what's available at: http://forge.puppetlabs.com/.

There's more

Not all publically available modules are on Puppet Forge. Some other great places to look are on GitHub: https://github.com/camptocamp https://github.com/example42 Dean Wilson maintains an excellent repository of Puppet patterns, tips, and recipes, at the Puppet Cookbook website: http://puppetcookbook.com/.

Puppet 2.7 Cookbook

Build reliable, scalable, secure, high-performance systems to fully utilize the power of cloud computing with this book and ebook

Published: October 2011
eBook Price: $26.99
Book Price: $44.99
See more

Select your format and quantity:

Creating your own resource types

It's time to get creative. You'll know about various different resource types in Puppet: packages, files, users, and so on. Usually, you can do everything you need to do by using either combinations of these built-in resources, or a custom define which you can use more or less in the same way as a resource. However, if you need to create your own resource type, Puppet makes it quite easy. The native types are written in Ruby, and you will need a basic familiarity with Ruby in order to create your own. Let's understand the distinction between types and providers. A type describes a resource and the parameters it can have (for example, the package type). A provider tells Puppet how to implement a resource for a particular platform or situation (for example, the apt/dpkg providers implement package for Debian-like systems). A single type (package) can have many providers (apt, yum, fink, and so on). If you don't specify a provider when declaring a resource, Puppet will choose the most appropriate one given the environment. In this section we'll see how to create a custom type to manage Git repositories, and in the next section, we'll write a provider to implement this type.

Getting ready

Enable pluginsync in your puppet.conf, if you haven't already:

[main]
pluginsync = true

Create a custom module for your plugins and types in your Puppet repository, if you haven't already:

# cd /etc/puppet/modules # mkdir custom

Within the module, create a lib/puppet/type directory:

# cd custom # mkdir -p lib/puppet/type

How to do it

Create a file in the type directory named gitrepo.rb with the following contents:

Puppet::Type.newtype(:gitrepo) do
    ensurable

    newparam(:source) do
        isnamevar
    end

    newparam(:path)
end

How it works...

The first line registers a new type named gitrepo:

Puppet::Type.newtype(:gitrepo) do

The ensurable line automatically gives the type a property ensure, like Puppet's built-in resources.

ensurable

We'll now give the type some parameters. For the moment, all we need is a source parameter for the Git source URL, and a path parameter to tell Puppet where the repository should be created in the filesystem.

newparam(:source) do
  isnamevar
end

The isnamevar declaration tells Puppet that the source parameter is the type's namevar. So when you declare an instance of this resource, whatever name you give it will be the value of source. For example:

gitrepo { "git://github.com/puppetlabs/puppet.git":
    path => "/home/john/work/puppet",
}

Finally, we add the path parameter:

newparam(:path)

There's more…

Once you're familiar with creating your own resources, you can use them to replace complicated exec resources and make your manifests more readable. However, it's a good idea to make your resources robust and reusable by adding some documentation, and validating your parameters.

Documentation

Our example is deliberately simple, but when you move on to developing real custom types for your production environment, you should add documentation strings to describe what the type and its parameters do. For example:

Puppet::Type.newtype(:gitrepo) do
    @doc = "Manages Git repos"

    ensurable

    newparam(:source) do
        desc "Git source URL for the repo"
        isnamevar
    end

    newparam(:path) do
        desc "Path where the repo should be created"
    end
end

Validation

You can use parameter validation to generate useful error messages when someone tries to pass bad values to the resource. For example, you could validate that the directory where the repository is to be created actually exists:

newparam(:path) do
     validate do |value|
         basepath = File.dirname(value)
         unless File.directory?(basepath)
           raise ArgumentError , "The path %s doesn't exist" 
           % basepath
         end
    end
end

You can also specify the list of allowed values that the parameter can take as follows:

newparam(:breakfast) do
    newvalues(:bacon, :eggs, :sausages) end

Using MCollective

The Marionette Collectiv (MCollective for short) is a tool for system administration. It can run commands on large numbers of servers in parallel, and uses a broadcast architecture so that you can administer a large network without the need for a central master server or asset database. Each server runs an MCollective daemon which listens for requests, and can execute commands locally or return information about the server. This can be used to filter the list of target servers. So, for example, you could use MCollective to execute a given command on all servers which match certain criteria. You can think of MCollective as a complement to Puppet (though it also works fine with Chef and other configuration management systems). For example, your provisioning process for a new node might require firewall changes on other machines, permissions granted on a database server, and so on, which is not very easy to do with Puppet. Although you could automate specific jobs using shell scripts and SSH, MCollective provides a powerful and flexible way to solve this general problem.

Getting ready

MCollective uses the ActiveMQ message broker framework (actually, any STOMP-compliant middleware, but ActiveMQ is a popular choice), which in turn requires Java, so if you don't have Java already installed on your system, install it:

# apt-get install gcj-4.4-jre-headless

Go to the ActiveMQ download page and get the latest stable "Unix distribution" tarball: http://activemq.apache.org/download.html.
Install the stomp gem as follows:

# gem install stomp

Download the latest stable MCollective .deb packages from: http://www.puppetlabs.com/misc/download-options/
Install the packages as follows:

# dpkg -i mcollective_1.0.1-1_all.deb mcollective-client_1.0.1-1_all.deb mcollective-common_1.0.1-1_all.deb

Download the tarball of the same release from the MCollective downloads page (because it contains an example ActiveMQ configuration file).
Edit the MCollective server.cfg file:
```
# vi /etc/mcollective/server.cfg
```
Set the plugin.stomp.host parameter to the name of your server (where you're running ActiveMQ):

plugin.stomp.host = cookbook.bitfieldconsulting.com

Make the same change in the MCollective client.cfg file:

# vi /etc/mcollective/client.cfg

Unpack the MCollective tarball and copy the example ActiveMQ configuration into place:

# tar xvzf mcollective-1.0.1.tgz # cp mcollective-1.0.1/ext/activemq/examples/single-broker/activemq.xml /etc/mcollective

Edit the configuration file to set the password of the mcollective user to the same as it is in server.cfg:

# vi /etc/mcollective/activemq.xml

Unpack the ActiveMQ tarball and start the server using the following config file:

# tar xvzf apache-activemq-5.4.2-bin.tar.gz # apache-activemq-5.4.2/bin/activemq start xbean:/etc/mcollective/activemq.xml INFO: Using default configuration (you can configure options in one of these file: /etc/default/activemq /root/.activemqrc) INFO: Invoke the following command to create a configuration file bin/activemq setup [ /etc/default/activemq | /root/.activemqrc ] INFO: Using java '/usr/bin/java' INFO: Starting - inspect logfiles specified in logging.properties and log4j.properties to get details INFO: pidfile created : '/root/apache-activemq-5.4.2/data/activemq.pid' (pid '3322')

Start the MCollective server:

# service mcollective start Starting mcollective: *

How to do it...

Check that MCollective and ActiveMQ are set up and working by running:

# mc-ping cookbook time=68.82 ms ---- ping statistics ---- 1 replies max: 68.82 min: 68.82 avg: 68.82

If you don't see any results, check that the mcollectived daemon is running, and that a Java process is also running for ActiveMQ.
Run mc-inventory against your machine to see what information MCollective knows about it:

# mc-inventory cookbook Inventory for cookbook: Server Statistics: Version: 1.0.1 Start Time: Mon Mar 07 11:44:53 -0700 2011 Config File: /etc/mcollective/server.cfg Process ID: 4220 Total Messages: 14 Messages Passed Filters: 6 Messages Filtered: 5 Replies Sent: 5 Total Processor Time: 0.8 seconds System Time: 0.47 seconds Agents: discovery rpcutil Configuration Management Classes: Facts: mcollective => 1

Create a new custom fact for the server by adding the following code snippet to /etc/mcollective/facts.yaml:

purpose: webserver

Now use MCollective to search for all machines matching this fact:

# mc-find-hosts --with-fact purpose=webserver cookbook

How it works...

MCollective is a broadcast framework; when you issue a request like mc-find-hosts, MCollective sends a message out to all clients asking, "Does anyone match this filter?" All clients that match the filter will send a reply, and MCollective gathers the results and prints them out for you. You can install a number of plugins and agents for specific tasks (for example, running Puppet). These are installed on the clients, and MCollective handles the communications involved in sending the command out to all matching machines, and collating any results

There's more…

Even though we've only taken a few steps with MCollective, it's clearly a powerful tool for both gathering information about servers, and executing commands on a list of servers which can be selected by facts. For example, you could get a list of all machines which haven't run Puppet in the last 24 hours. Or, you could take some action on all webservers, or all machines with an x86_64 architecture. MCollective itself only provides a framework for such applications. There are a variety of plugins available which do useful things, and writing your own plugins is easy.

Summary

In this article we covered a few useful topics such as using Public modules from Puppet Forge,creating your own resource types and Using MCollective.

Further resources on this subject:

Creating Your First Virtual Machine: Ubuntu Linux (Part 1) [Article]
Creating Your First Virtual Machine: Ubuntu Linux (Part 2) [Article]
Change Control for Personal Projects - Subversion Style [Article]

Puppet 2.7 Cookbook

Build reliable, scalable, secure, high-performance systems to fully utilize the power of cloud computing with this book and ebook

Published: October 2011
eBook Price: $26.99
Book Price: $44.99
See more

Select your format and quantity:

About the Author :

John Arundel

John Arundel is an infrastructure consultant who helps people make their computer systems more reliable, useful, and cost-effective and has fun doing it. He has what Larry Wall describes as the three great virtues of a programmer: laziness, impatience, and hubris.

Laziness, because he doesn't like doing work that a computer could do instead. Impatience, because he wants to get stuff done right away. Hubris, because he likes building systems that are as good as he can make them.

He was formerly a senior operations engineer at global telco Verizon, designing resilient, high-performance infrastructures for corporations such as Ford, McDonald's, and Bank of America. He now works independently, helping to bring enterprise-grade performance and reliability to clients with slightly smaller pockets but very big ideas.

He likes writing books, especially about Puppet. It seems that at least some people enjoy reading them, or maybe they just like the pictures. He also occasionally provides training and coaching on Puppet, which turns out to be far harder than simply doing the work himself.

Off the clock, he can usually be found driving a Land Rover up some mountain or other. He lives in a small cottage in Cornwall and believes, like Cicero, that if you have a garden and a library, then you have everything you need.

You can follow him on Twitter at @bitfield.