Puppet: Integrating External Tools

Exclusive offer: get 50% off this eBook here
Puppet 2.7 Cookbook

Puppet 2.7 Cookbook — Save 50%

Build reliable, scalable, secure, high-performance systems to fully utilize the power of cloud computing with this book and ebook

$26.99    $13.50
by John Arundel | November 2011 | Cookbooks Open Source

Puppet is a useful tool by itself, but you can get much greater benefits from using Puppet in combination with other tools and frameworks. We'll take a look at some of these.

In this article by John Arundel, author of Puppet 2.7 Cookbook,we will cover the following topics:

  • Executing commands before and after Puppet runs
  • Using Public modules
  • Creating your own resource types
  • Using MCollective

Executing commands before and after Puppet runs

If you need to have a command executed before each Puppet run, you can do this using the prerun_command configuration setting. Similarly, you can use postrun_command to execute a command after the run has completed. This mechanism gives you a powerful hook to integrate Puppet with other software, or even trigger events on other machines. The prerun and postrun commands must succeed (that is, return a zero exit status), or Puppet will report an error. This enables you to have any command failures reported using Puppet's reporting mechanism, for example.

How to do it...

Set prerun_command or postrun_command in puppet.conf to the commands you want to run:

prerun_command = /usr/local/bin/before-puppet-run.sh postrun_command = /usr/local/bin/after-puppet-run.sh

There's more

You can use prerun and postrun commands to integrate Puppet with Ubuntu's etckeeper system. Etckeeper is a version control system for tracking changes to files in the /etc directory. To do this, define these commands in puppet.conf:

prerun_command=/etc/puppet/etckeeper-commit-pre postrun_command=/etc/puppet/etckeeper-commit-post

Using public modules

"Plagiarize, plagiarize, plagiarize / Only be sure always to call it please 'research' "—Tom Lehrer, 'Lobachevsky'

If in doubt, steal. In many cases when you write a Puppet module to manage some software or service, you don't have to start from scratch. Community-contributed modules are available at the Puppet Forge site for many popular applications. Sometimes, a community module will be exactly what you need and you can download and start using it straight away. In other cases, you will need to make some modifications to suit your particular needs and environment. If you are new to Puppet, it can be a great help to have some existing code to start with. On the other hand, community modules are often written to be as general and portable as possible, and the extra code required can make them harder to understand. In general I would not recommend treating Puppet Forge as a source of 'drop-in' modules which you can deploy without reading or understanding the code. This introduces an external dependency to your Puppet infrastructure, and doesn't help advance your understanding and experience of Puppet. Rather, I would use it as a source of inspiration, help, and examples. A module taken from Puppet Forge should be a jumping-off point for you to develop and improve your own modules. Be aware that a given module may not work on your Linux distribution. Check the README file which comes with the module to see if your operating system is supported.

Getting ready

  1. The easiest way to use Puppet Forge modules is to install the puppet-module tool:
  2. # gem install puppet-module Fetching: puppet-module-0.3.2.gem (100%) ****************************************************************************** Thank you for installing puppet-module from Puppet Labs! * Usage instructions: read "README.markdown" or run `puppet-module usage` * Changelog: read "CHANGES.markdown" or run `puppet-module changelog` * Puppet Forge: visit http://forge.puppetlabs.com/ ****************************************************************************** Successfully installed puppet-module-0.3.2 1 gem installed Installing ri documentation for puppet-module-0.3.2... Installing RDoc documentation for puppet-module-0.3.2...

  3. Run puppet-module to see the available commands:
  4. # puppet-module Tasks: puppet-module build [PATH_TO_MODULE] # Build a module for release puppet-module changelog # Display the changelog for this tool puppet-module changes [PATH_TO_MODULE] # Show modified files in an installed m... puppet-module clean # Clears module cache for all repositories puppet-module generate USERNAME-MODNAME # Generate boilerplate for a new module puppet-module help [TASK] # Describe available tasks or one speci... puppet-module install MODULE_NAME_OR_FILE [OPTIONS] # Install a module (eg, 'user-modname')... puppet-module repository # Show currently configured repository puppet-module search TERM # Search the module repository for a mo... puppet-module usage # Display detailed usage documentation ... puppet-module version # Show the version information for this... Options: -c, [--config=CONFIG] # Configuration file # Default: /etc/puppet/puppet.conf

How to do it

In this example, we'll use puppet-module to find and install a module to manage the Tomcat application server.

  1. Search for a suitable module as follows:
  2. # puppet-module search tomcat ===================================== Searching http://forge.puppetlabs.com ------------------------------------- 2 found. -------- camptocamp/tomcat (0.0.1) jeffmccune/tomcat (1.0.1)

  3. In this example we'll install the Jeff McCune version:
  4. # cd /etc/puppet/modules # puppet-module install jeffmccune/tomcat Installed "jeffmccune-tomcat-1.0.1" into directory: jeffmccune-tomcat

  5. The module is now ready to use in your manifests: looking at the source code will show you how to do this.

How it works...

The puppet-module tool simply automates the process of searching and downloading modules from the Puppet Forge site. You can browse the site to see what's available at: http://forge.puppetlabs.com/.

There's more

Not all publically available modules are on Puppet Forge. Some other great places to look are on GitHub: https://github.com/camptocamp https://github.com/example42 Dean Wilson maintains an excellent repository of Puppet patterns, tips, and recipes, at the Puppet Cookbook website: http://puppetcookbook.com/.

Puppet 2.7 Cookbook Build reliable, scalable, secure, high-performance systems to fully utilize the power of cloud computing with this book and ebook
Published: October 2011
eBook Price: $26.99
Book Price: $44.99
See more
Select your format and quantity:

Creating your own resource types

It's time to get creative. You'll know about various different resource types in Puppet: packages, files, users, and so on. Usually, you can do everything you need to do by using either combinations of these built-in resources, or a custom define which you can use more or less in the same way as a resource. However, if you need to create your own resource type, Puppet makes it quite easy. The native types are written in Ruby, and you will need a basic familiarity with Ruby in order to create your own. Let's understand the distinction between types and providers. A type describes a resource and the parameters it can have (for example, the package type). A provider tells Puppet how to implement a resource for a particular platform or situation (for example, the apt/dpkg providers implement package for Debian-like systems). A single type (package) can have many providers (apt, yum, fink, and so on). If you don't specify a provider when declaring a resource, Puppet will choose the most appropriate one given the environment. In this section we'll see how to create a custom type to manage Git repositories, and in the next section, we'll write a provider to implement this type.

Getting ready

  1. Enable pluginsync in your puppet.conf, if you haven't already:
  2. [main] pluginsync = true

  3. Create a custom module for your plugins and types in your Puppet repository, if you haven't already:
  4. # cd /etc/puppet/modules # mkdir custom

  5. Within the module, create a lib/puppet/type directory:
  6. # cd custom # mkdir -p lib/puppet/type

How to do it

Create a file in the type directory named gitrepo.rb with the following contents:

Puppet::Type.newtype(:gitrepo) do ensurable newparam(:source) do isnamevar end newparam(:path) end

How it works...

The first line registers a new type named gitrepo:

Puppet::Type.newtype(:gitrepo) do

The ensurable line automatically gives the type a property ensure, like Puppet's built-in resources.

ensurable

We'll now give the type some parameters. For the moment, all we need is a source parameter for the Git source URL, and a path parameter to tell Puppet where the repository should be created in the filesystem.

newparam(:source) do isnamevar end

The isnamevar declaration tells Puppet that the source parameter is the type's namevar. So when you declare an instance of this resource, whatever name you give it will be the value of source. For example:

gitrepo { "git://github.com/puppetlabs/puppet.git": path => "/home/john/work/puppet", }

Finally, we add the path parameter:

newparam(:path)

There's more…

Once you're familiar with creating your own resources, you can use them to replace complicated exec resources and make your manifests more readable. However, it's a good idea to make your resources robust and reusable by adding some documentation, and validating your parameters.

Documentation

Our example is deliberately simple, but when you move on to developing real custom types for your production environment, you should add documentation strings to describe what the type and its parameters do. For example:

Puppet::Type.newtype(:gitrepo) do @doc = "Manages Git repos" ensurable newparam(:source) do desc "Git source URL for the repo" isnamevar end newparam(:path) do desc "Path where the repo should be created" end end

Validation

You can use parameter validation to generate useful error messages when someone tries to pass bad values to the resource. For example, you could validate that the directory where the repository is to be created actually exists:

newparam(:path) do validate do |value| basepath = File.dirname(value) unless File.directory?(basepath) raise ArgumentError , "The path %s doesn't exist" % basepath end end end

You can also specify the list of allowed values that the parameter can take as follows:

newparam(:breakfast) do
    newvalues(:bacon, :eggs, :sausages) 

end

Using MCollective

The Marionette Collectiv (MCollective for short) is a tool for system administration. It can run commands on large numbers of servers in parallel, and uses a broadcast architecture so that you can administer a large network without the need for a central master server or asset database. Each server runs an MCollective daemon which listens for requests, and can execute commands locally or return information about the server. This can be used to filter the list of target servers. So, for example, you could use MCollective to execute a given command on all servers which match certain criteria. You can think of MCollective as a complement to Puppet (though it also works fine with Chef and other configuration management systems). For example, your provisioning process for a new node might require firewall changes on other machines, permissions granted on a database server, and so on, which is not very easy to do with Puppet. Although you could automate specific jobs using shell scripts and SSH, MCollective provides a powerful and flexible way to solve this general problem.

Getting ready

  1. MCollective uses the ActiveMQ message broker framework (actually, any STOMP-compliant middleware, but ActiveMQ is a popular choice), which in turn requires Java, so if you don't have Java already installed on your system, install it:
  2. # apt-get install gcj-4.4-jre-headless

  3. Go to the ActiveMQ download page and get the latest stable "Unix distribution" tarball: http://activemq.apache.org/download.html.
  4. Install the stomp gem as follows:
  5. # gem install stomp

  6. Download the latest stable MCollective .deb packages from: http://www.puppetlabs.com/misc/download-options/
  7. Install the packages as follows:
  8. # dpkg -i mcollective_1.0.1-1_all.deb mcollective-client_1.0.1-1_all.deb mcollective-common_1.0.1-1_all.deb

  9. Download the tarball of the same release from the MCollective downloads page (because it contains an example ActiveMQ configuration file).
  10. Edit the MCollective server.cfg file:

    # vi /etc/mcollective/server.cfg

  11. Set the plugin.stomp.host parameter to the name of your server (where you're running ActiveMQ):
  12. plugin.stomp.host = cookbook.bitfieldconsulting.com

  13. Make the same change in the MCollective client.cfg file:
  14. # vi /etc/mcollective/client.cfg

  15. Unpack the MCollective tarball and copy the example ActiveMQ configuration into place:
  16. # tar xvzf mcollective-1.0.1.tgz # cp mcollective-1.0.1/ext/activemq/examples/single-broker/activemq.xml /etc/mcollective

  17. Edit the configuration file to set the password of the mcollective user to the same as it is in server.cfg:
  18. # vi /etc/mcollective/activemq.xml

  19. Unpack the ActiveMQ tarball and start the server using the following config file:
  20. # tar xvzf apache-activemq-5.4.2-bin.tar.gz # apache-activemq-5.4.2/bin/activemq start xbean:/etc/mcollective/activemq.xml INFO: Using default configuration (you can configure options in one of these file: /etc/default/activemq /root/.activemqrc) INFO: Invoke the following command to create a configuration file bin/activemq setup [ /etc/default/activemq | /root/.activemqrc ] INFO: Using java '/usr/bin/java' INFO: Starting - inspect logfiles specified in logging.properties and log4j.properties to get details INFO: pidfile created : '/root/apache-activemq-5.4.2/data/activemq.pid' (pid '3322')

  21. Start the MCollective server:

    # service mcollective start Starting mcollective: *

How to do it...

  1. Check that MCollective and ActiveMQ are set up and working by running:
  2. # mc-ping cookbook time=68.82 ms ---- ping statistics ---- 1 replies max: 68.82 min: 68.82 avg: 68.82

  3. If you don't see any results, check that the mcollectived daemon is running, and that a Java process is also running for ActiveMQ.
  4. Run mc-inventory against your machine to see what information MCollective knows about it:
  5. # mc-inventory cookbook Inventory for cookbook: Server Statistics: Version: 1.0.1 Start Time: Mon Mar 07 11:44:53 -0700 2011 Config File: /etc/mcollective/server.cfg Process ID: 4220 Total Messages: 14 Messages Passed Filters: 6 Messages Filtered: 5 Replies Sent: 5 Total Processor Time: 0.8 seconds System Time: 0.47 seconds Agents: discovery rpcutil Configuration Management Classes: Facts: mcollective => 1

  6. Create a new custom fact for the server by adding the following code snippet to /etc/mcollective/facts.yaml:
  7. purpose: webserver

  8. Now use MCollective to search for all machines matching this fact:
  9. # mc-find-hosts --with-fact purpose=webserver cookbook

How it works...

MCollective is a broadcast framework; when you issue a request like mc-find-hosts, MCollective sends a message out to all clients asking, "Does anyone match this filter?" All clients that match the filter will send a reply, and MCollective gathers the results and prints them out for you. You can install a number of plugins and agents for specific tasks (for example, running Puppet). These are installed on the clients, and MCollective handles the communications involved in sending the command out to all matching machines, and collating any results

There's more…

Even though we've only taken a few steps with MCollective, it's clearly a powerful tool for both gathering information about servers, and executing commands on a list of servers which can be selected by facts. For example, you could get a list of all machines which haven't run Puppet in the last 24 hours. Or, you could take some action on all webservers, or all machines with an x86_64 architecture. MCollective itself only provides a framework for such applications. There are a variety of plugins available which do useful things, and writing your own plugins is easy.

Summary

In this article we covered a few useful topics such as using Public modules from Puppet Forge,creating your own resource types and Using MCollective.


Further resources on this subject:


Puppet 2.7 Cookbook Build reliable, scalable, secure, high-performance systems to fully utilize the power of cloud computing with this book and ebook
Published: October 2011
eBook Price: $26.99
Book Price: $44.99
See more
Select your format and quantity:

About the Author :


John Arundel

John Arundel is a devops consultant, which means he solves difficult problems for a living. (He doesn't get called in for easy problems.)

He has worked in the tech industry for 20 years, and during that time has done wrong (or seen done wrong) almost everything that you can do wrong with computers. That comprehensive knowledge of what not to do, he feels, is one of his greatest assets as a consultant. He is still adding to it.

He likes writing books, especially about Puppet (The Puppet 3 Beginner's Guide is available from the same publisher). It seems that at least some people enjoy reading them. He also provides training and coaching on Puppet, which it turns out is far harder than simply doing the work himself.

Off the clock, he can usually be found driving a Land Rover up some mountain or other. He lives in a small cottage in Cornwall and believes, like Cicero, that if you have a garden and a library, then you have everything you need.

You can follow him on Twitter at @bitfield.

Books From Packt

 

PHP Team Development
PHP Team Development

CFEngine 3 Beginner’s Guide
CFEngine 3 Beginner’s Guide

Linux Shell Scripting Cookbook
Linux Shell Scripting Cookbook

Nginx HTTP Server
Nginx HTTP Server

 Nginx 1 Web Server Implementation Cookbook
Nginx 1 Web Server Implementation Cookbook

MySQL Admin Cookbook
MySQL Admin Cookbook

 Drupal 7 Social Networking
Drupal 7 Social Networking

Koha 3 Library Management System
Koha 3 Library Management System


No votes yet

Post new comment

CAPTCHA
This question is for testing whether you are a human visitor and to prevent automated spam submissions.
6
k
R
z
V
q
Enter the code without spaces and pay attention to upper/lower case.
Code Download and Errata
Packt Anytime, Anywhere
Register Books
Print Upgrades
eBook Downloads
Video Support
Contact Us
Awards Voting Nominations Previous Winners
Judges Open Source CMS Hall Of Fame CMS Most Promising Open Source Project Open Source E-Commerce Applications Open Source JavaScript Library Open Source Graphics Software
Resources
Open Source CMS Hall Of Fame CMS Most Promising Open Source Project Open Source E-Commerce Applications Open Source JavaScript Library Open Source Graphics Software