Mastering Puppet 5

Using a decent text editor with the plugins that equip you to write well for Puppet is a really  good step toward better quality. There are quite a few options out there, and it's best to use whatever suits your own unique writing style. Personally, I have used Atom (https://atom.io) most successfully, and recently installed it locally on my workstation. I used Eclipse many years ago (this has also been known previously as Geppetto), which I in fact felt was unwieldy due to a large memory footprint. It's also nice to remain fairly handy with Vim, especially for working on the command line server-side, or if you use a Linux OS on your workstation. There's also TextMate, for an macOS X only editor that has all of Apple's look and feel.

Let's take a look at some of the various options for an Integrated Development Machine (IDE) available to us as Puppet developers.

cd ~
git clone https://github.com/ricciocri/vimrc .vim
cd .vim
git pull && git submodule init && git submodule update && git submodule status
cd ~
ln -s .vim/.vimrc

$ mkdir ~/temp
$ cd ~/temp
$ git clone https://github.com/masterzen/puppet-textmate-bundle.git Puppet.tmbundle
$ mv ~/temp/Puppet.tmbundle ~/Library/Application\ Support/TextMate/Bundles/
$ rm -fr ~/temp

This section contains a set of recommendations surrounding good module and class design. Bear in mind that Puppet development is, in principle, just like any other type of software development, and we've learned over many years in software development, and especially at O&O software, that certain modular and class design principles make our development better. I also feel that part of our journey toward infrastructure as code is making our Puppet code just as well-designed, structured, and tested as any other application code.

include mymodule

class {'mymodule':
  myparam => false,
}

mymodule::mydefine {‘define1':
  myotherparam => false,
}

class vhost (
  $servername,
  $serveraliases,
  $port
)
{ ...

class vhost (
  String  $servername,
  Array   $serveraliases,
  Integer $port
)
{ ...

Some features to improve quality in your Puppet development, such as puppet-lint, puppet-rspec, and commands such as puppet module create have been around for some time, but previously, you had to discover these tools out there in the wild, install them, and figure out how to use them effectively yourself.

Puppet decided back in August 2017 to bring these things all together on the client side and make them a breeze to use with the new Puppet Development Kit version 1.0. I can certainly recall puppet-rspec always took some time to set up and get working correctly. Now it's all really easy.

Let's take a whistle-stop tour of the module development process using the new PDK 1.0.

$ pdk new module zope –-skip-interview

So, just use the name of the module to create init.pp:

$ pdk new class zope

These commands now negate any need for snippets in your text editor to create the comments, declarations, and other boilerplate code.

$ pdk new class params

As you are working, you can use the new pdk validate command (https://puppet.com/docs/pdk/1.0/pdk_reference.html#pdk-validate-command) to assist with checking that the module compiles, conforms to the Puppet Style Guide, and has valid metadata:

$ pdk validate

The number one most important thing you can do to bring quality to your modules is to test them! Testing really is one of the most important aspects of software quality assurance in any field of software development. In the agile development community, we've been banging on the table about automated testing for more than 10 years!

Puppet RSpec (http://rspec-puppet.com/tutorial) has been allowing the Puppet community to unit test their modules for quite some time, but it's even easier now with the new PDK 1.0, as everything is set up ready, and you can just add your testing code and run the tests.

From a Puppet perspective, unit testing means checking the output from the compiler. Are the resources contained in the compiled relationship resource catalog, and is their order as expected, given the parameters passed and/or facts present?

When you begin to write tests in Puppet-RSpec, it seems at first like all you are doing is rewriting the Puppet manifests in another Ruby-like language. There is, however, really more to it than that. If there is some reasonable complexity to the module's functionality, for example, testing the dynamic content produced by Puppet templates, support for multiple operating systems, and different actions according to the passed parameters, then these tests actually form a safety net when editing or adding new functionality to your modules, protecting against regressions when refactoring, or upgrading to a new Puppet release.

Let's carry on from the previous two sections and use the development kit to unit test our module. Whenever you generate a class using the pdk new class command, PDK creates a corresponding unit test file. This file, located in your module's /spec/classes folder, already includes a template for writing your unit tests (see http://rspec-puppet.com/tutorial). You can then run the tests using the following command:

$ pdk test unit

Be on the lookout for code smells, especially as your Puppet code base ages! The following link is a research project that describes a bunch of Puppet code smells, which is an XP (extreme programming) term meaning code issues—usually meaning either a poor design or implementation: http://www.tusharma.in/wp-content/uploads/2016/03/ConfigurationSmells_preprint.pdf

Let's quickly run through using the Puppeteer Python-based tool used in the preceding research project:

$ git clone https://github.com/tushartushar/Puppeteer
$ cd Puppeteer

Another issue that can often hit you as your Puppet code base ages is unused code in your codebase. But, there's a tool out there in the wild we can use to keep on top of this issue.

puppet-ghostbuster essentially compares what is actually being used (stored in PuppetDB) to what you think you are using (in your code base directory). This give you the opportunity to slash and burn anything that's really unused. This is great from the point of view of software maintainability. A smaller code base is simply cheaper to maintain!

Let's quickly run through using this Ruby gem.

Make the following settings in your environment variables:

Run the command as follows:

$ find . -type f -exec puppet-lint --only-checks ghostbuster_classes,ghostbuster_defines,ghostbuster_facts,ghostbuster_files,ghostbuster_functions,ghostbuster_hiera_files,ghostbuster_templates,ghostbuster_types {} \+

You can add to and remove from the comma-delimited items to check for unused classes, defined types, facts, files, functions, Hiera files, templates, and types.

It maybe goes without saying that there's no reason to reinvent the wheel when you are authoring your Puppet modules. A few minutes in Puppet Forge (https://forge.puppet.com) can really save you days and days of editing. There are, at the time of writing, more than 5,000 Forge modules, so it makes a great deal of sense to leverage all that hard work done by the Puppet community. Search the Forge first for that bit of software; it's more than likely that something already exists.

In my experience, I have found there is often something that almost does the job. Maybe there's a module (usually an unsupported and unapproved one) that maybe, for example, performs the management for the software you require, but it's only for Ubuntu, and you're using Red Hat. It's usually a better approach to fork that module, whatever shape it's in, and work on that, rather than start from scratch.

The best way for me to describe this best practice is to use an anti-pattern as an example.

I once came across a Puppet developer who would start a module completely from scratch, and then copy and paste lines of code from a Forge module into the new module. From then on, that module exists entirely outside the community! It's not a fork even, so to integrate changes that have been made over time from the community becomes a real pain. You would have to cherry-pick those changes to get the functionality into your own, and you will probably still be left with regression problems. Generally, a best practice is to always at the very least fork the Forge module! This means you get the Git history, which often contains the thoughts that have gone into producing that module.

You see, if you were ever a reader of the great book The Cathedral & the Bazaar: Musings on Linux and Open Source by an Accidental Revolutionary (https://www.amazon.com/Cathedral-Bazaar-Musings-Accidental-Revolutionary/dp/0596001088), then you will understand that the Linux-orientated philosophy of software development through a bazaar, collaborative working style trumps spinning off development into a cathedral, independent working style. Well, that's my take on this developer's working style. He was working cathedral-ly, as opposed to bazaar-ly. Effectively, you are making the decision to pit your cathedral team against the multitude of the bazaar, and to my mind, that's simply not wise project management when it comes to giving you a competitive advantage in the internet age.

Sometimes, modules on the Forge get a bit out of date. If the metadata for the module is out of date, you can always produce that again using the PDK new module command (https://puppet.com/docs/pdk/1.0/pdk_generating_modules.html#create-a-module-with-pdk) and commit the new metadata.

Of course, to be a great Puppet community member, it would be an even better practice to make pull requests for the changes you have made and contribute to the work of the community.

Another important recommendation is to simply write great documentation. There's nothing worse, I feel, as a developer, than to have to dig into the code to understand how a module works; it's like having to lift the hood of the car to understand how to drive a vehicle!

Get good at writing English to convey technical ideas! I really think it's a skill that every good developer really needs to master.

Edit the module's metadata.json file to add module dependencies. See the following example:

"dependencies": [
    { "name":" stankevich/python",
       "version_requirement":">= 1.18.x"
     }
  ]

The name key is the name of the requirement, namely, "pe" or "puppet". The version_requirement key is a semver (http://semver.org) value or range. See the following examples:

These would all be valid values for version_requirement.

Check the metadata.json file for validity afterwards using the new PDK command, as follows:

$ pdk validate metadata

The great thing about adding module dependencies is the fact that, when you run the puppet module download command, Puppet will download all the module dependencies accordingly.

This section introduces you to adding compatibility data for the module designed for your version of Puppet or Puppet Enterprise and the operating system you want to work with. To begin with, Edit the module's metadata.json file to add compatibility data.

"operatingsystem_support": [
       { "operatingsystem": "RedHat", },
       { "operatingsystem": "Ubuntu", },
]

"operatingsystem_support": [
       {
           "operatingsystem":"RedHat",
           "operatingsystemrelease":[ "5.0", "6.0" ]
       },
       {
           "operatingsystem": "Ubuntu",
           "operatingsystemrelease": [
               "12.04",
               "10.04"
           ]
       }
   ]

$ pdk validate metadata

"requirements": [ {“name”: “pe”, “version_requirement”: “5.x”}]

$ pdk validate metadata

For quite some time when module writing, we've been using the params.pp pattern. One class in the module, by convention called <MODULENAME>::params, sets the variables for any of the other classes:

class zope::params {
  $autoupdate = false,
  $default_service_name = 'ntpd',


  case $facts['os']['family'] {
    'AIX': {
      $service_name = 'xntpd'
    }
    'Debian': {
      $service_name = 'ntp'
    }
    'RedHat': {
      $service_name = $default_service_name
    }
  }
}

So, you can see here that we are using some conditional logic depending on the os::family fact, so that the service_name variable can be set appropriately. We are also exposing the autoupdate variable, and giving it a default value.

This params.pp pattern is an elegant little hack, which takes advantage of Puppet's idiosyncratic class inheritance behavior (using inheritance is generally not recommended in Puppet). Then, any of the other classes in the module inherit from the params class, to have their parameters set appropriately, as shown in the following example:

class zope (
  $autoupdate   = $zope::params::autoupdate,
  $service_name = $zope::params::service_name,
) inherits zope::params {
 ...
}

Since the release of Hiera 5, we are able to simplify our module complexity considerably. By using Hiera-based defaults, we can simplify our module's main classes, and they no longer need to inherit from params.pp. Additionally, you no longer need to explicitly set a default value with the = operator in the parameter declaration.

Let's look at the equivalent configuration to the params.pp pattern using Hiera 5.

First of all, in order to use this new functionality, the data_provider key needs to be set to the heiravalue in the module's metadata.json file:

...
"data_provider": "hiera",
...

Next, we need to add a hiera.yaml file to the root directory of the module:

---
version: 5
defaults:
  datadir: data
  data_hash: yaml_data
hierarchy:
  - name: "OS family"
    path: "os/%{facts.os.family}.yaml"


  - name: "common"
    path: "common.yaml"

We can then add three files to the /data directory (note that the datadir setting in the hiera.yaml file). The first file of these three is used to set the AIX service_name variable:

# zope/data/os/AIX.yaml
---
zope::service_name: xntpd

The second file is used to set the Debian service_name variable:

# zope/data/os/Debian.yaml
zope::service_name: ntp

And finally, there is the common file, and Hiera will fall through to this file to find its values if it doesn't find a corresponding operating system file when looking for the service_name setting, or a value for autoupdate when searching the previous two files:

# ntp/data/common.yaml
---
ntp::autoupdate: false
ntp::service_name: ntpd

We will look at Hiera 5 in much more detail in Chapter 4, Hiera 5.

In this chapter, we have covered a lot of ground, and I've introduced a bunch of best practices you can use to produce better quality component modules.

In the next chapter, we'll still be covering development in Puppet DSL, and turn our attention to two special modules: role and profile, which can help us to build reusable, configurable, and refactorable site-wide configuration code.