Separation of Code and Data with Hiera

Working through the first seven chapters, you have used the basic structural elements of Puppet in numerous examples and contexts. There has been a quick demonstration of the more advanced language features, and you should have a good idea of what distinguishes the manifest writing process in Puppet 4 from those of the earlier releases.

For all their expressive power, manifests do have some limitations. A manifest that is designed by the principles taught up to this point mixes logic with data. Logic is not only evident in control structures, such as if and else, but it also emerges from the network of classes and defines that include and instantiate one another.

However, you cannot configure a machine by just including some generic classes. Many properties of a given system are individual and must be passed as parameters. This can have maintenance...

Understanding the need for separate data storage

Looking back at what you have implemented during this book so far, you have managed to create some very versatile code that did very useful things in an automatic fashion. Your nodes can distribute entries for /etc/hosts among themselves. They register each other's public SSH key for authentication. A node can automatically register itself to a central Cacti server.

Thanks to Facter, Puppet has the information that allows the effortless handling of these use cases. Many configuration items are unique to each node only because they refer to a detail (such as an IP address or a generated key) that is already defined. Sometimes, the required configuration data can only be found on a remote machine, which Puppet handles through exported resources. Such manifest designs that can rely on facts are very economical. The information...

Building hierarchical data structures

In the previous section, we reduced the data problem to a simple need for key-value pairs that are specific to each node under Puppet management. Puppet and its manifests then serve as the engine that generates actual configuration from these minimalistic bits of information.

A simplistic approach to this problem is an ini style configuration file that has a section for each node that sets values for all configurable keys. Shared values will be declared in one or more general sections:

[mysql]
buffer_pool=15G
log_file_size=500M
...
[xndp12-sql01.example.net]
psk=xneFGl%23ndfAWLN34a0t9w30.zges4
server_id=1

Rails applications customarily do something similar and store their configuration in a YAML format. The user can define different environments, such as production, staging, and testing. The values that are defined per environment override the global...

Fetching data from classes

Looking up a key value in Hiera is easy. Puppet comes with a very straightforward function for this:

$plugins = hiera('reporting::plugins') 

Whenever the compiler encounters such a call in the manifest of the current agent node, it triggers a search in the hierarchy. The specific data sources are determined by the hierarchy in your hiera.yaml file. It will almost always rely on fact values provided by the agent to make flexible data source selections.

If the named key cannot be found in the agent's hierarchy, the master aborts the catalog compilation with an error. To prevent this, it is often sensible to supply a default value with the lookup:

$plugins = hiera('reporting::plugins', []) 

In this case, Puppet uses an empty array if the hierarchy mentions no plugins.

On the other hand, you can purposefully omit the default value...

Debugging data lookups

As you can see from the preceding example, the data that contributes to the complete configuration of any module can be rather dispersed throughout the set of your data sources. It can be challenging to determine where the respective values are retrieved from for any given agent node. It can be frustrating to trace data sources to find out why a change at some level will not take effect for some of your agents.

To help make the process more transparent, Hiera comes with a command-line tool called hiera. Invoking it is simple:

root@puppetmaster # hiera -c /etc/puppetlabs/code/hiera.yaml demo::atoms  

It retrieves a given key using the specified configuration from hiera.yaml. Make sure that you use the same Hiera configuration as Puppet.

Of course, this can only work sensibly if Hiera selects the same data sources as the compiler, which uses fact values to...

Managing resources from data

You can now move configuration settings to Hiera and dedicate your manifest to logic. This works seamlessly as far as classes and their parameters are concerned, because class parameters automatically retrieve their values from Hiera. For configuration that requires you to instantiate resources, you still need to write the full manifests and add manual lookup function calls.

For example, an Apache web server requires some global settings, but the interesting parts of its configuration are typically performed in virtual host configuration files. Puppet models them with defined resource types. If you want to configure an iptables firewall, you have to declare lots of resources of the firewall type (available through the puppetlabs-firewall module).

Such elaborate resources can clutter up your manifest, yet they mostly represent data. There is no inherent...

Summary

Hiera is a tool that stores and retrieves data in a hierarchical fashion. Each retrieval uses a distinct data source from each hierarchy layer and traverses your hierarchy from the most to the least specific level. The hierarchy is defined by the user as an array in a YAML file.

Puppet has Hiera support built in, and you can use it to separate data from code. From manifests, you will mainly perform lookups through the hiera function. In most cases, the respective entries will rely on fact values.

Another common way to employ Hiera through Puppet is to name the Hiera keys in the <class-name>::<parameter-name> format. When including a parameterized class, Puppet will look for such keys in Hiera. If the manifest does not supply a parameter value, Puppet automatically binds the value from Hiera to the respective parameter.

Manifests that boast large numbers of...