In this article by Jussi Heinonen, the author of Learning Puppet, we will get started with creating a Puppet module and the various aspects associated with it.

Together with all the manifest files that we created so far, there are several of them already, and we haven't yet started to develop Puppet manifests. As the number of manifests expand, one may start wondering how files can be distributed and applied efficiently across multiple systems.

This article will introduce you to Puppet modules and show you how to prepare a simple web server environment with Puppet.

(For more resources related to this topic, see here.)

Introducing the Puppet module

The Puppet module is a collection of code and data that usually solves a particular problem, such as the installation and configuration of a web server. A module is packaged and distributed in the TAR (tape archive) format. When a module is installed, Puppet extracts the archive file on the disk, and the output of the installation process is a module directory that contains Puppet manifests (code), static files (data), and template files (code and data).

Static files are typically some kind of configuration files that we want to distribute across all the nodes in the cluster. For example, if we want to ensure that all the nodes in the cluster are using the same DNS server configuration, we can include the /etc/resolv.conf file in the module and tell Puppet to apply it across all the nodes. This is just an example of how static files are used in Puppet and not a recommendation for how to configure DNS servers.

Like static files, template files can also be used to provide configuration. The difference between a static and template file is that a static file will always have the same static content when applied across multiple nodes, whereas the template file can be customized based on the unique characteristics of a node. A good example of a unique characteristic is an IP address. Each node (or a host) in the network must have a unique IP address. Using the template file, we can easily customize the configuration on every node, wherever the template is applied.

It's a good practice to keep the manifest files short and clean to make them easy to read and quick to debug. When I write manifests, I aim to keep the length of the manifest file in less than a hundred lines. If the manifest length exceeds 100 lines, then this means that I may have over-engineered the process a little bit. If I can't simplify the manifest to reduce the number of lines, then I have to split the manifest into multiple smaller manifest files and store these files within a Puppet module.

The Puppet module structure

The easiest way to get familiar with a module structure is to create an empty module with the puppet module generate command. As we are in the process of building a web server that runs a web application, we should give our module a meaningful name, such as learning-webapp.

The Puppet module name format

Before we create our first module, let's take a quick look at the Puppet module naming convention. The Puppet module name is typically in the format of <author>-<modulename>. A module name must contain one hyphen character (no more, no less) that separates the <author> and the <modulename> names. In the case of our learning-webapp module that we will soon create, the author is called learning and the module name is webapp, thus the module name learning-webapp.

Generating a Puppet module

Let's take a look at the following steps to create the learning-webapp Puppet module:

1. Start the puppet-agent virtual machine.
2. Using the cd command, navigate to the directory that is shared via the shared folder.
3. On my virtual machine, my shared folder appears as /media/sf_learning, and I can move to the directory by running the following command:

   # cd /media/sf_learning

4. Then, I'll create an empty puppet module with the command puppet module generate learning-webapp --skip-interview and the command returns a list of files and directories that the module contains:

   # puppet module generate learning-webapp --skip-interview
   Notice: Generating module at /media/sf_learning/learning-webapp
   Notice: Populating templates...
   Finished; module generated in learning-webapp.
   learning-webapp/metadata.json
   learning-webapp/Rakefile
   learning-webapp/manifests
   learning-webapp/manifests/init.pp
   learning-webapp/spec
   learning-webapp/spec/spec_helper.rb
   learning-webapp/spec/classes
   learning-webapp/spec/classes/init_spec.rb
   learning-webapp/Gemfile
   learning-webapp/tests
   learning-webapp/tests/init.pp
   learning-webapp/README.md
   

5. To get a better view of how the files in the directories are organized in the learning-webapp module, you can run the tree learning-webapp command, and this command will produce the following tree structure of the files:

    

Here, we have a very simple Puppet module structure. Let's take a look at the files and directories inside the module in more detail:

• Gemfile: A file used for describing the Ruby package dependencies that are used for unit testing.

  For more information on Gemfile, visit http://bundler.io/v1.3/man/gemfile.5.html.

• manifests: A directory for all the Puppet manifest files in the module.
• manifests/init.pp: A default manifest file that declares the main Puppet class called webapp.
• metadata.json: A file that contains the module metadata, such as the name, version, and module dependencies.
• README.md: A file that contains information about the usage of the module.
• Spec: An optional directory for automated tests.
• Tests: A directory that contains examples that show how to call classes that are stored in the manifests directory.
• tests/init.pp: A file containing an example how to call the main class webapp in file manifests/init.pp.

A Puppet class

A Puppet class is a container for Puppet resources. A class typically includes references to multiple different types of resources and can also reference other Puppet classes.

The syntax for declaring a Puppet class is not that different from declaring Puppet resources. A class definition begins with the keyword class, followed by the name of the class (unquoted) and an opening curly brace ({). A class definition ends with a closing curly brace (}).

Here is a generic syntax of the Puppet class:

class classname {
}

Let's take a look at the manifests/init.pp file that you just created with the puppet module generate command. Inside the file, you will find an empty Puppet class called webapp. You can view the contents of the manifests/init.pp file using the following command:

# cat /media/sf_learning/learning-webapp/manifests/init.pp

The init.pp file mostly contains the comment lines, which are prefixed with the # sign, and these lines can be ignored. At the end of the file, you can find the following declaration for the webapp class:

class webapp {
}

The webapp class is a Puppet class that does nothing as it has no resources declared inside it.

Resources inside the Puppet class

Let's add a notify resource to the webapp class in the manifests/init.pp file before we go ahead and apply the class. The notify resource does not manage any operating system resources, such as files or users, but instead, it allows Puppet to report a message when a resource is processed.

As the webapp module was created inside shared folders, you no longer have to use the Nano editor inside the virtual machine to edit manifests. Instead, you can use a graphical text editor, such as a Notepad on Windows or Gedit on the Linux host. This should make the process of editing manifests a bit easier and more user friendly.

The directory that I shared on the host computer is /home/jussi/learning. When I take a look inside this directory, I can find a subdirectory called learning-webapp, which is the Puppet module directory that we created a moment ago. Inside this, there is a directory called manifests, which contains the init.pp file.

Open the init.pp file in the text editor on the host computer and scroll down the file until you find the webapp class code block that looks like the following:

class webapp {
}

If you prefer to carry on using the Nano editor to edit manifest files (I salute you!), you can open the init.pp file inside the virtual machine with the nano /media/sf_learning/learning-webapp/manifests/init.pp command.

The notify resource that we are adding must be added inside the curly braces that begins and ends the class statement; otherwise, the resource will not be processed when we apply the class.

Now we can add a simple notify resource that makes the webapp class look like the following when completed:

class webapp {
notify { 'Applying class webapp':
}
}

Let's take a look at the preceding lines one by one:

• Line 1 begins with the webapp class, followed by the opening curly brace.
• Line 2 declares a notify resource and a new opening curly brace, followed by the resource name. The name of the notify resource will become the message that Puppet prints on the screen when the resource from a class is processed.
• Line 3 closes the notify resource statement.
• Line 4 indicates that the webapp class finishes here.

Once you have added the notify resource to the webapp class, save the init.pp file.

Rename the module directory

Before we can apply our webapp class, we must rename our module directory. It is unclear to me as to why the puppet module generate command creates a directory name that contains a hyphen character (as in learning-webapp). The hyphen character is not allowed to be present in the Puppet module directory name. For this reason, we must rename the learning-webapp directory before we can apply the webapp class inside it.

As the learning-webapp module directory lives in the shared folders, you can either use your preferred file manager program to rename the directory, or you can run the following two commands inside the Puppet Learning VM to change the directory name from learning-webapp to webapp:

# cd /media/sf_learning
# mv learning-webapp webapp

Your module directory name should now be webapp, and we can move on to apply the webapp class inside the module and see what happens.

Applying a Puppet class

You can try running the puppet apply webapp/manifests/init.pp command but don't be disappointed when nothing happens. Why is that?

The reason is because there is nothing inside the init.pp file that references the webapp class. If you are familiar with object-oriented programming, you may know that a class must be instantiated in order to get services from it. In this case, Puppet behaves in a similar way to object-oriented programming languages, as you must make a reference to the class in order to tell Puppet to process the class.

Puppet has an include keyword that is used to reference a class. The include keyword in Puppet is only available for class resources, and it cannot be used in conjunction with any other type of Puppet resources.

To apply the webapp class, we can make use of the init.pp file under the tests directory that was created when the module was generated. If you take a look inside the tests/init.pp file, you will find a line include webapp. The tests/init.pp file is the one that we should use to apply the webapp class.

Here are the steps on how to apply the webapp class inside the Puppet Learning VM:

1. Go to the parent directory of the webapp module:

   # cd /media/sf_learning

2. Apply the webapp class that is included in the tests/init.pp file:

   # puppet apply --modulepath=./ webapp/tests/init.pp

3. When the class is applied successfully, you should see the notify resource that was added to the webapp class that appears on lines 2 and 3 in the following Puppet report:

   Notice: Compiled catalog for web.development.vm in environment production in 0.05 seconds
   Notice: Applying class webapp
   Notice: /Stage[main]/Webapp/Notify[Applying class webapp]/message: defined 'message' as 'Applying class webapp'
   Notice: Finished catalog run in 0.81 seconds
   

Let's take a step back and look again at the command that we used to apply to the webapp class:

# puppet apply --modulepath=./ webapp/tests/init.pp

The command can be broken down into three elements:

• puppet apply: The puppet apply command is used when applying a manifest from the command line.
• modulepath=./: This option is used to tell Puppet what filesystem path to use to look for the webapp module. The ./ (dot forward slash) notation means that we want our current /media/sf_learning working directory to be used as the modulepath value.
• webapp/tests/init.pp: This is the file that the puppet apply command should read.

Installing a module from Puppet Forge

Puppet Forge is a public Puppet module repository (https://forge.puppetlabs.com) for modules that are created by the community around Puppet. Making use of the modules in Puppet Forge is a great way to build a software stack quickly, without having to write all the manifests yourself from scratch.

The web server that we are going to install is a highly popular Apache HTTP Server (http://httpd.apache.org/), and there is a module in Puppet Forge called puppetlabs-apache that we can install. The Puppetlabs-apache module provides all the necessary Puppet resources for the Apache HTTP Server installation.

Note that the puppet module installation requires an Internet connection. To test whether the Puppet Learning VM can connect to the Internet, run the following command on the command line:

# host www.google.com

On successful completion, the command will return the following output:

www.google.com has address 216.58.211.164
www.google.com has IPv6 address 2a00:1450:400b:801::2004

Note that the reported IP address may vary. As long as the host command returns www.google.com has address …, the Internet connection works.

Now that the Internet connection has been tested, you can now proceed with the module installation.

Before we install the puppetlabs-apache module, let's do a quick search to confirm that the module is available in Puppet Forge. The following command will search for the puppetlabs-apache module:

# puppet module search puppetlabs-apache

When the search is successful, it returns the following results:

Then, we can install the module. Follow these steps to install the puppetlabs-apache module:

1. In the Puppet Learning VM, go to the shared folders /media/sf_learning directory by running the cd /media/sf_learning command.
2. Then, run the following command:

   # puppet module install --modulepath=./ puppetlabs-apache

   The --modulepath=./ option specifies that the module should be installed in the current /media/sf_learning working directory

3. The installation will take a couple of minutes to complete, and once it is complete, you will see the following lines appear on the screen:

   Notice: Preparing to install into /media/sf_learning ...

   Notice: Preparing to install into /media/sf_learning ...
   Notice: Downloading from https://forgeapi.puppetlabs.com ...
   Notice: Installing -- do not interrupt ...
   /media/sf_learning
   └─┬ puppetlabs-apache (v1.2.0)
      ├── puppetlabs-concat (v1.1.2)
      └── puppetlabs-stdlib (v4.8.0)
   

Let's take a look at the output line by line to fully understand what happened during the installation process:

• Line 1 tells us that the module is going to be installed in the /media/sf_learning directory, which is our current working directory. This directory was specified with the --modulepath=./ option in the puppet module install command.
• Line 2 says that the module is going to be installed from https://forgeapi.puppetlabs.com/, which is the address for Puppet Forge.
• Line 3 is fairly self-explanatory and indicates that the installation process is running.
• Lines 4 and 5 tell us that the puppetlabs-apache module was installed in the current /media/sf_learning working directory.
• Line 6 indicates that as part of the puppetlabs-apache module installation, a puppetlabs-concat dependency module was also installed.
• Line 7 lists another dependency module called puppetlabs-stdlib that got installed in the process.

Now you can run the tree -L 1 command to see what new directories got created in /media/sf_learning as a result of the puppet module install command:

# tree -L 1
├── apache
├── concat
├── stdlib
└── webapp
4 directories, 0 files

The argument -L 1 in the tree command specifies that it should only traverse one level of directory hierarchy.

Installing Apache HTTP Server

Now that the puppetlabs-apache module is installed in the filesystem, we can proceed with the Apache HTTP Server installation.

Earlier, we talked about how a Puppet class can be referenced with the include keyword. Let's see how this works in practice by adding the include apache statement to our webapp class, and then applying the webapp class from the command line.

Open the webapp/manifests/init.pp file in your preferred text editor, and add the include apache statement inside the webapp class.

I like to place the include statements at the beginning of the class before any resource statement. In my text editor, the webapp class looks like the following after the include statement has been added to it:

Once you have saved the webapp/manifests/init.pp file, you can apply the webapp class with the following command:

# puppet apply --modulepath=./ webapp/tests/init.pp

This time, the command output is much longer compared to what it was when we applied the webapp class for the first time. In fact, the output is too long to be included in full, so I'm only going to show you the last two lines of the Puppet report, which shows you the step where the state of the Service[httpd] resource has changed from stopped to running:

Notice: /Stage[main]/Apache::Service/Service[httpd]/ensure: ensure changed 'stopped' to 'running'Notice: Finished catalog run in 65.20 seconds

Summary

So we have now come to the end of this article. I hope you found the content useful and not too challenging. One of the key deliverables of this article was to experiment with Puppet modules and learn how to create your own module