Mastering Puppet

In our example installation, we will be using the name puppet.example.com for our Puppet server. Starting with a server installation of Enterprise Linux version 6, we install httpd and mod_ssl using the following code:

# yum install httpd mod_ssl
Installed:
  httpd-2.2.15-29.el6_4.x86_64
  mod_ssl-2.2.15-29.el6_4.x86_64

To install mod_passenger, we pull in the Extra Packages for Enterprise Linux (EPEL) repository available at https://fedoraproject.org/wiki/EPEL. Install the EPEL repository by downloading the rpm file from http://download.fedoraproject.org/pub/epel/6/x86_64/repoview/epel-release.html or use the following code:

# yum install http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
Installed:
  epel-release-6-8.noarch

Once EPEL is installed, we install mod_passenger from that repository using the following code:

# yum install mod_passenger
Installed:
  mod_passenger-3.0.21-5.el6.x86_64

Next, we will pull in Puppet from the puppetlabs repository available at http://docs.puppetlabs.com/guides/puppetlabs_package_repositories.html#for-red-hat-enterprise-linux-and-derivatives using the following code:

# yum install http://yum.puppetlabs.com/el/6/products/x86_64/puppetlabs-release-6-7.noarch.rpm
Installed:
  puppetlabs-release-6-7.noarch

With the puppetlabs repository installed, we can then install Puppet using the following command:

# yum install puppet
Installed:
  puppet-3.3.2-1.el6.noarch

The Puppet rpm will create the /etc/puppet and /var/lib/puppet directories. In /etc/puppet, there will be a template puppet.conf; we begin by editing that file to set the name of our Puppet server (puppet.example.com) in the certname setting using the following code:

[main]
  logdir = /var/log/puppet
  rundir = /var/run/puppet
  vardir = /var/lib/puppet
  ssldir = $vardir/ssl
  certname = puppet.example.com
  [agent]
  server = puppet.example.com
  classfile = $vardir/classes.txt
  localconfig = $vardir/localconfig

The other lines in this file are defaults. At this point, we would expect puppet.example.com to be resolved with a DNS query correctly, but if you do not control DNS at your organization or cannot have this name resolved properly at this point, edit /etc/hosts, and put in an entry for your host pointing to puppet.example.com. In all the examples, you would substitute example.com for your own domain name.

127.0.0.1   localhost localhost.localdomain puppet 
  puppet.example.com

We now need to create certificates for our master; to ensure the Certificate Authority (CA) certificates are created, run Puppet cert list using the following command:

# puppet cert list
Notice: Signed certificate request for ca

In your enterprise, you may have to answer requests from multiple DNS names, for example, puppet.example.com, puppet, and puppet.devel.example.com. To make sure our certificate is valid for all those DNS names, we will pass the dns-alt-names option to puppet certificate generate; we also need to specify that the certificates are to be signed by the local machine using the following command:

puppet# puppet certificate generate --ca-location local --dns-alt-names puppet,puppet.prod.example.com,puppet.dev.example.com puppet.example.com
Notice: puppet.example.com has a waiting certificate request
true

Now, to sign the certificate request, first verify the certificate list using the following commands:

puppet# puppet cert list
  "puppet.example.com" (SHA256) E5:F7:26:0A:6C:41:26:FA:80:02:E5:A6:A1:DB:F4:E0:9D:9C:5B:2D:A5:BF:EC:D1:FA:84:51:F4:8C:FD:9B:AF (alt names: "DNS:puppet", "DNS:puppet.dev.example.com", "DNS:puppet.example.com", "DNS:puppet.prod.example.com")
puppet# puppet cert sign puppet.example.com
Notice: Signed certificate request for puppet.example.com
Notice: Removing file Puppet::SSL::CertificateRequest puppet.example.com at '/var/lib/puppet/ssl/ca/requests/puppet.example.com.pem'

$ puppet config print ssldir

One last task is to copy the certificate that you just signed into certs by navigating to /var/lib/puppet/ssl/certs. You can use Puppet certificate find to do this using the following command:

# puppet certificate find puppet.example.com --ca-location local
-----BEGIN CERTIFICATE-----
MIIF1TCCA72gAwIBAgIBAjANBgkqhkiG9w0BAQsFADAoMSYwJAYDVQQDDB1QdXBw
...
-----END CERTIFICATE-----

When you install Puppet from the puppetlabs repository, the rpm will create an Apache configuration file called apache2.conf. Locate this file and copy it into your Apache configuration directory using the following command:

# cp /usr/share/puppet/ext/rack/example-passenger-vhost.conf /etc/httpd/conf.d/puppet.conf

We will now show the Apache config file and point out the important settings using the following configuration:

PassengerHighPerformance on
PassengerMaxPoolSize 12
PassengerPoolIdleTime 1500
# PassengerMaxRequests 1000
PassengerStatThrottleRate 120
RackAutoDetect Off
RailsAutoDetect Off

The preceding lines of code configure passenger for performance. PassengerHighPerformance turns off some compatibility that isn't required. The other options are tuning parameters. For more information on these settings, see http://www.modrails.com/documentation/Users%20guide%20Apache.html.

Next we will need to modify the file to ensure it points to the newly created certificates. We will need to edit the lines for SSLCertificateFile and SSLCertificateKeyFile. The other SSL file settings should point to the correct certificate, chain, and revocation list files as shown in the following code:

Listen 8140
<VirtualHost *:8140>
  ServerName puppet.example.com
  SSLEngine on
  SSLProtocol -ALL +SSLv3 +TLSv1
  SSLCipherSuite ALL:!ADH:RC4+RSA:+HIGH:+MEDIUM:-LOW:-SSLv2:-EXP

  SSLCertificateFile /var/lib/puppet/ssl/certs/puppet.example.com.pem
  SSLCertificateKeyFile /var/lib/puppet/ssl/private_keys/puppet.example.com.pem
  SSLCertificateChainFile /var/lib/puppet/ssl/ca/ca_crt.pem
  SSLCACertificateFile /var/lib/puppet/ssl/ca/ca_crt.pem
  # If Apache complains about invalid signatures on the CRL, you can try disabling
  # CRL checking by commenting the next line, but this is not recommended.
  SSLCARevocationFile /var/lib/puppet/ssl/ca/ca_crl.pem
  SSLVerifyClient optional
  SSLVerifyDepth 1
  # The `ExportCertData` option is needed for agent certificate expiration warnings
  SSLOptions +StdEnvVars +ExportCertData
  RequestHeader set X-SSL-Subject %{SSL_CLIENT_S_DN}e
  RequestHeader set X-Client-DN %{SSL_CLIENT_S_DN}e
  RequestHeader set X-Client-Verify %{SSL_CLIENT_VERIFY}e

  DocumentRoot /etc/puppet/rack/public/
  RackBaseURI /
<Directory /etc/puppet/rack/>
  Options None
  AllowOverride None
  Order allow,deny
  allow from all
</Directory>
</VirtualHost>

In this VirtualHost we listen on 8140 and configure the SSL certificates in the SSL lines. The RequestHeader lines are used to pass certificate information to the Puppet process spawned by passenger. The DocumentRoot and RackBaseURI settings are used to tell passenger where to find its configuration file config.ru. We create /etc/puppet/rack and it's subdirectories and then copy the example config.ru into that directory using the following commands:

# mkdir -p /etc/puppet/rack/{public,tmp}
# cp /usr/share/puppet/ext/rack/files/config.ru /etc/puppet/rack
# chown puppet:puppet /etc/puppet/rack/config.ru

We change the owner of config.ru to puppet:puppet as the passenger process will run as the owner of config.ru. Our config.ru will contain the following code:

$0 = "master"

# if you want debugging:
# ARGV << "--debug"

ARGV << "--rack"
ARGV << "--confdir" << "/etc/puppet"
ARGV << "--vardir"  << "/var/lib/puppet"

require 'puppet/util/command_line'
run Puppet::Util::CommandLine.new.execute

The config.ru file sets the command-line arguments for Puppet. The ARGV lines are used to set additional parameters to the puppet process. As noted in the Puppet master main page, any valid configuration parameter from puppet.conf can be specified as an argument here. Only the options that affect where Puppet will look for files should be specified here. Once puppet knows where to find puppet.conf, adding arguments here could be confusing.

With this configuration in place, we are ready to start Apache as our Puppet master. Simply start Apache with a service httpd start.

# setenforce 0 
# service auditd rotate
# service httpd restart
(start a puppet run remotely)
# audit2allow -i /var/log/audit/audit.log -M puppet_passenger
# semodule -i puppet_passenger.pp
# setenforce 1

Now that Puppet is running, you'll need to open the local firewall (iptables) on port 8140 to allow your nodes to connect. Then you'll need an example site.pp to get started. For testing we will create a basic site.pp that defines a default node with a single class attached to the default node as shown in the following code:

node default {
  include example
}

class example {
  notify {"This is an example": }
}

You can start a practice node or two and run their agent against the Puppet server either using --server puppet.example.com or editing the agents puppet.conf file to point at your server. Agents will by default look for an unqualified host called Puppet. Then search based on your DNS configuration (search in /etc/resolv.conf), and if you do not control DNS, you may have to edit the local /etc/hosts file to specify the IP address of your Puppet master. A sample run, for a node called node1, should look something like the following commands:

[root@node1 ~]# puppet agent -t
Info: Creating a new SSL key for node1
Info: Caching certificate for ca
Info: Creating a new SSL certificate request for node1
Info: Certificate Request fingerprint (SHA256): C4:0D:7A:54:ED:C8:E8:CC:68:D0:A6:13:C4:91:28:3D:B1:66:71:48:57:85:D8:99:AF:D0:81:54:B9:64:AB:F2
Exiting; no certificate found and waitforcert is disabled

Sign the certificate on the Puppet master and run again; the run should look like the following commands:

[root@puppet ~]# puppet cert sign node1
Notice: Signed certificate request for node1
Notice: Removing file Puppet::SSL::CertificateRequest node1 at '/var/lib/puppet/ssl/ca/requests/node1.pem'

[root@node1 ~]# puppet agent -t
Info: Caching certificate for node1
Info: Caching certificate_revocation_list for ca
Info: Retrieving plugin
Info: Caching catalog for node1
Info: Applying configuration version '1386310193'
Notice: This is an example
Notice: /Stage[main]/Example/Notify[This is an example]/message: defined 'message' as 'This is an example'
Notice: Finished catalog run in 0.03 seconds

You now have a working passenger configuration. This configuration can handle a much larger load than the default WEBrick server provided with puppet. Puppet Labs suggests the WEBrick server is appropriate for small installations; in my experience that number is much less than 100 nodes, maybe even less than 50. You can tune the passenger configuration and handle a large number of nodes, but to handle a very large installation (1000s of nodes), you'll need to start splitting up the workload.

Puppet is a web service. But there are several different components supporting that web service, as shown in the following diagram:

Each of the different components in your Puppet infrastructure: SSL CA, Reporting, Storeconfigs, and Catalog compilation can be split up into their own server or servers.

Catalog compilation

Catalog compilation is the one task that can really bog down your Puppet installation. Splitting compilation among a pool of workers is the biggest win for scaling your deployment. The idea here is to have a primary point of contact for all your nodes—the Puppet master. Then, using proxying techniques, the master will direct requests to specific worker machines within your Puppet infrastructure. From the perspective of the nodes checking into the Puppet master, all the interaction appears to come from the main proxy machine.

To understand how we are going to achieve this load balancing, we first need to look at how the agents request data from our Puppet master. The request URL sent to our Puppet master has the format https://puppetserver:8140/environment/resource/key. The "environment" in the request URL is the Puppet environment in use by the node. It defaults to production but can be other values as we will see in later chapters. The resource being requested can be any of the accepted REST API calls, such as: catalog, certificate, resource, report, file_metadata, or file_content. A complete listing of the http_api is available at http://docs.puppetlabs.com/guides/rest_api.html.

Requests from nodes to the Puppet masters follow a pattern that we can use to configure our proxy machine. The pattern is as follows:

/environment/resource/key

For example, when node1.example.com requests its catalog in the production environment, it connects to the server and requests the following (using URL encoding):

https://puppet.example.com:8140/production/catalog/node1.example.com.

Knowing that there is a pattern to the requests, we can configure Apache to redirect requests based on regular expression matches to different machines in our Puppet infrastructure.

Our first step in splitting up our load will be to clone our Puppet master server twice to create two new worker machines, which we will call worker1.example.com and worker2.example.com. In this example, we will use 192.168.100.101 for worker1 and 192.168.100.102 for worker2. Create a private network for all the Puppet communication on 192.168.100.0/24. Our Puppet master will use the address 192.168.100.100. It is important to create a private network for the worker machines as our proxy configuration removes the SSL encryption, which means that communication between the workers and the master proxy machine is unencrypted.

Our new Puppet infrastructure is shown in the following diagram:

On our Puppet server, we will change the Apache puppet.conf as follows. Instead of listening on 8140, we will listen on 18140, and importantly, only listen on our private network as this traffic will be unencrypted. Next, we will not enable SSL on 18140. And finally we will remove any header settings we were making in our original file as shown in the following configuration:

PassengerHighPerformance on
PassengerMaxPoolSize 12
PassengerPoolIdleTime 1500
# PassengerMaxRequests 1000
PassengerStatThrottleRate 120
RackAutoDetect Off
RailsAutoDetect Off

Listen 127.0.0.1:18140
Listen 192.168.100.100:18140

<VirtualHost *:18140>
  ServerName puppet.example.com
  DocumentRoot /etc/puppet/rack/public/
  RackBaseURI /
  <Directory /etc/puppet/rack/>
    Options None
    AllowOverride None
    Order allow,deny
    allow from all
  </Directory>
</VirtualHost>

The configuration for this VirtualHost is much simpler. Now, on the worker machines, create /etc/httpd/conf.d/puppet.conf files that are identical to the previous files but have different Listen directives shown as follows:

• On worker1:

  Listen 192.168.100.101:18140
  

• On worker2:

  Listen 192.168.100.102:18140
  

Remember to open port 18140 on the worker machines' firewalls (iptables) and start httpd.

Returning to the Puppet master machine, create a proxy.conf file in the Apache conf.d directory (/etc/httpd/conf.d) to point at the workers. We will create two proxy pools. The first is for certificate signing, called puppetca, as shown in the following configuration:

<Proxy balancer://puppetca>
BalancerMember http://127.0.0.1:181
40
</Proxy>

A second proxy pool is for catalog compilation, called puppetworker, as shown in the following configuration:

<Proxy balancer://puppetworker>
BalancerMember http://192.168.100.102:181
40
BalancerMember http://192.168.100.101:181
40
</Proxy>

Next recreate the Puppet VirtualHost listener for 8140 with the SSL and certificate information used previously, as shown in the following configuration:

LoadModule ssl_module modules/mod_ssl.so

Listen 8140
<VirtualHost *:8140>
ServerName puppet.example.com
       SSLEngine on
       SSLProtocol -ALL +SSLv3 +TLSv1
       SSLCipherSuite ALL:!ADH:RC4+RSA:+HIGH:+MEDIUM:-LOW:-SSLv2:-
       EXP
       SSLCertificateFile 
       /var/lib/puppet/ssl/certs/puppet.example.com.pem
       SSLCertificateKeyFile 
  /var/lib/puppet/ssl/private_keys/puppet.example.com.pem
       SSLCertificateChainFile /var/lib/puppet/ssl/ca/ca_crt.pem
       SSLCACertificateFile    /var/lib/puppet/ssl/ca/ca_crt.pem
       # If Apache complains about invalid signatures on the CRL, you can try disabling
       # CRL checking by commenting the next line, but this is not recommended.
       SSLCARevocationFile     /var/lib/puppet/ssl/ca/ca_crl.pem
       SSLVerifyClient optional
       SSLVerifyDepth  1
       # The `ExportCertData` option is needed for agent certificate expiration warnings
       SSLOptions +StdEnvVars +ExportCertData
       # This header needs to be set if using a loadbalancer or proxy
       RequestHeader unset X-Forwarded-For
       RequestHeader set X-SSL-Subject %{SSL_CLIENT_S_DN}e
       RequestHeader set X-Client-DN %{SSL_CLIENT_S_DN}e
       RequestHeader set X-Client-Verify %{SSL_CLIENT_VERIFY}e

Since we know that we want all certificate requests going to the puppetca balancer, we use ProxyPassMatch to match URLs that have a certificate as the second phrase following the environment as shown in the next configuration. Our regular expression searches for a single word followed by /certificate.*, and any match is sent to our puppetca balancer.

ProxyPassMatch ^/([^/]+/certificate.*)$ balancer://puppetca/$1

The only thing that remains is to send all noncertificate requests to our load balancing pair, worker1 and worker2, as shown in the following configuration:

ProxyPass / balancer://puppetworker/
ProxyPassReverse / balancer://puppetworker
</VirtualHost>

At this point, we can restart Apache on the Puppet master.

Tip

SELinux

You'll need to allow Puppet to bind to port 18140 at this point since the default puppet SELinux module allows for 8140 only. You will also need to allow Apache to connect to the worker instances; there is a Boolean for that, httpd_can_network_connect.

Now, when a node connects, if it requests for a certificate, it will be redirected to the VirtualHost on port 18140 on the Puppet master. If the node requests a catalog, it will be redirected to one of the worker nodes. To convince yourself that this is the case, edit /etc/puppet/manifests/site.pp on your worker1 node and insert notify as shown in the following configuration:

node default {
  include example
  notify {'Compiled on worker1': }
}

Do the same on worker2 with the message Compiled on worker2, run puppet agent again on your node, and see where the catalog is being compiled using the following commands:

[root@node1 ~]# puppet agent –t
Info: Retrieving plugin
Info: Caching catalog for node1
Info: Applying configuration version '1386312527'
Notice: Compiled on worker1
Notice: /Stage[main]//Node[default]/Notify[Compiled on worker1]/message: defined 'message' as 'Compiled on worker1'
Notice: This is an example
Notice: /Stage[main]/Example/Notify[This is an example]/message: defined 'message' as 'This is an example'
Notice: Finished catalog run in 0.10 seconds

Tip

You may see "Compiled on worker2", which is expected.

To verify that certificates are being handled properly, clean the certificate for your example node, remove it from the node, and restart the agent.

• On the master:

  master# puppet cert clean node1
  

• On the node:

  node1# \rm -r /var/lib/puppet/ssl/*
  node1# puppet agent -t
  

Tip

Alternatively to this configuration, you could use the puppetca setting in puppet.conf on your nodes to get clients to use a specific machine for signing requests.

Since this is an enterprise installation, we should have a dashboard of some kind running to collect reports from workers.

Tip

If your reports setting on the master is either HTTP or puppetdb, then this section won't affect you.

We'll clone our worker again to make a new server called reports (192.168.100.103), which will collect our reports. We then have to add another line to our Apache proxy.conf configuration file to use the new server, and we need to place this line directly after the certificate proxy line. Since reports must all be sent to the same machine to be useful, we won't use a balancer line as before, and we will simply set the proxy to the address of the reports machine directly.

ProxyPassMatch ^/([^/]+/certificate.*)$ balancer://puppetca/$1
ProxyPassMatch ^/([^/]+/report
/.*)$ http://192.168.100.103/$1
ProxyPass / balancer://puppetworker/

Keep the /etc/httpd/conf.d/proxy.conf balancer section updated to send reports to 192.168.100.103.

Again, restart Apache and make sure that report=true is set on the node in the [agent] section of puppet.conf. Run Puppet agent on the node, and verify that the report gets sent to 192.168.100.103 (look in /var/lib/puppet/reports/).

Tip

If you are still seeing problems with client catalog compilation timeouts after creating multiple catalog workers, it may be that your client is timing out the connection before the worker has a chance to compile the catalog. Try experimenting with the configtimeout parameter in the [agent] section of puppet.conf

configtimeout=300

Setting this higher may resolve your issue. You will need to change the ProxyTimeout directive in the proxy.conf configuration for Apache as well. This will be revisited in Chapter 10, Troubleshooting.

puppet# ssh-keygen -f puppet_rsync
(creates puppet_rsync.pub puppet_rsync)

worker1# mkdir /var/lib/puppet/.ssh
# cp puppet_rsync.pub /var/lib/puppet/.ssh/authorized_keys
# chown -R puppet:puppet /var/lib/puppet/.ssh
# chmod 700 /var/lib/puppet/.ssh
# chmod 600 /var/lib/puppet/.ssh/authorized_keys
# chsh -s /bin/bash puppet

puppet# rsync -e 'ssh -i puppet_rsync' -az /etc/puppet/ puppet@worker1:/etc/puppet

Now that we have our Puppet infrastructure running on two workers and the master, you might notice that the main Apache virtual machine need not be on the same machine as the certificate-signing machine. At this point, there is no need to run passenger on that main gateway machine, and you are open to use whatever load balancing solution you see fit. In this example I will be using nginx as the main proxy point.

The important thing to remember here is that we are just providing a web service. We'll intercept the SSL part of the communication with nginx and then forward it onto our worker and CA machines as necessary. Our configuration will now look like the following diagram:

We will start with a blank machine this time; we do not need to install passenger or Puppet on the machine. To make use of the latest SSL-handling routines, we will download nginx from the nginx repository.

# yum install http://nginx.org/packages/rhel/6/noarch/RPMS/nginx-release-rhel-6-0.el6.ngx.noarch.rpm
Installed:
  nginx-release-rhel.noarch 0:6-0.el6.ngx
# yum install nginx
Installed:
  nginx-1.4.4-1.el6.ngx.x86_64

Now we need to copy the SSL CA files from the Puppet master to this gateway using the following commands:

puppet# scp /var/lib/puppet/ssl/ca/ca_crl.pem gateway:/etc/nginx
puppet# scp /var/lib/puppet/ssl/ca/ca_crt.pem gateway:/etc/nginx
puppet# scp /var/lib/puppet/ssl/certs/puppet.example.com.pem gateway:/etc/nginx
puppet# scp /var/lib/puppet/ssl/private_keys/puppet.example.com.pem gateway:/etc/nginx/puppet.example.com.key

Now we need to create a gateway configuration for nginx, which we will place in /etc/ngninx/conf.d/puppet-proxy.conf

We will define the two proxy pools as we did before, but using nginx syntax this time.

upstream puppetca {
  server 192.168.100.100:18140;
}

upstream puppetworkers {
  server 192.168.100.101:8140;
  server 192.168.100.102:8140;
}

Next, we create a server stanza, specifying that we handle the SSL connection, and we need to set some headers before passing on the communication to our proxied servers.

server {
  listen 8140 ssl;
  server_name puppet.example.com;

  default_type application/x-raw;

  ssl on;
  ssl_certificate    puppet.example.com.pem;
  ssl_certificate_key  puppet.example.com.key;
  ssl_trusted_certificate  ca_crt.pem;
  ssl_crl      ca_crl.pem;

  ssl_session_cache  shared:SSL:5m;
  ssl_session_timeout  5m;

  ssl_protocols    SSLv2 SSLv3 TLSv1;
  ssl_ciphers  
  ALL:!ADH:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP;
  ssl_prefer_server_ciphers on;
  ssl_verify_client optional_no_ca;

Setting ssl_verify_client to optional_no_ca is important, since on the first connection, the client will not have a signed certificate, so we need to accept all connections but mark a header with the verification status.

  proxy_set_header  Host      $host;
  proxy_set_header  X-Real-IP  $remote_addr;
  proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;
  proxy_set_header  X-Client-Verify  $ssl_client_verify;
  proxy_set_header  X-Client-DN    $ssl_client_s_dn;
  proxy_set_header  X-SSL-Subject    $ssl_client_s_dn;
  proxy_set_header   X-SSL-Issuer    $ssl_client_i_dn;
  proxy_read_timeout  1000;

The header X-Client-Verify will hold success or failure at this point, so our Puppet master will know if the certificate is valid. Now we need to look for certificate requests and hand those off to the puppetca pool:

location ~* ^/.*/certificate {
  proxy_pass http://puppetca;
  proxy_redirect off;
  proxy_read_timeout 1000;
}

Then we can send all other requests to our worker pool

location / {
  proxy_pass http://puppetworkers;
  proxy_redirect off;
  proxy_read_timeout 1000;
}

Now we need to start nginx on the gateway machine, open up port 8140 on the firewall, and open up 18140 on the Puppet master firewall (gateway will now need to communicate with that port).

Running puppet again on your node will now produce the same results as before, but you are now able to leverage the load balancing of nginx over that of Apache.

We have already split our workload into a certificate-signing machine (the master or puppetca), a pool of catalog machines, and a report-gathering machine. What is interesting as an exercise at this point is that we can also serve files up using our gateway machine.

Based on what we know about the puppet HTTP API, we know that requests for file_buckets, and files have specific URIs that we can serve directly from nginx without using passenger or Apache or even puppet. To test the configuration, alter the definition of the example class to include a file as follows:

class example {
  notify { 'This is an example': }
  file {'/tmp/example':
    mode => 644, 
    owner => 100,
    group => 100,
    source => 'puppet:///modules/example/example',
  }
}

Create the example file in /etc/puppet/modules/example/files/example.

This file lives on the workers. On the gateway machine, rsync your Puppet module code from the workers into /var/lib/nginx/puppet. Now, to prove that the file is coming from the gateway, edit the example file after you run the rsync.

The /etc/puppet/modules/example/files/example file lives on the gateway. At this point, we can start serving up files from nginx by putting in a location clause as follows; we will do two stanzas, one for files outside modules and the other for module-provided files at /etc/nginx/conf.d/gateway.conf.

location ~* ^/.*/file_content/modules {
  rewrite ^/([^/]+)/file_content/modules/([^/]+)/(.*) /$2/files/$3;
  break;
  root /var/lib/nginx/puppet/modules/;
}
location ~* ^/.*/file_content/ {
  rewrite ^/([^/]+)/file_content/([^/]+)/(.*) /$2/files/$3;
  break;
  root /var/lib/nginx/puppet/;
}

Restart nginx on the gateway machine, and then run Puppet on the node using the following command:

[root@node1 ~]# puppet agent –t
…
Notice: /Stage[main]/Example/File[/tmp/example]/ensure: defined content as '{md5}c83849f23a139c41edfbcd8473a81ac1'
…
Notice: Finished catalog run in 0.16 seconds
[root@node1 ~]# cat /tmp/example
This file lives on the gateway

As we can see, although the file living on the workers has the contents "This file lives on the workers," our node is getting the file directly from nginx on the gateway.

One important thing to consider is security, as any configured client can retrieve files from our gateway machine. In production, you would want to add ACLs to the file location.

As we have seen, once the basic proxying is configured, further splitting up of the workload becomes a routine task. We can split the workload to scale to handle as many nodes as we require.

To start our rpm, we will make an rpm spec file, we can make this anywhere since we don't have a master in this example. Start by installing rpm-build, which will allow us to build the rpm.

# yum install rpm-build
Installing
  rpm-build-4.8.0-37.el6.x86_64

It will be important later to have a user to manage the repository, so create a user called builder at this point. We'll do this on the Puppet master machine we built earlier. Create an rpmbuild directory with the appropriate subdirectories, and then create our example code in this location.

# sudo -iu builder
$ mkdir -p rpmbuild/{SPECS,SOURCES}
$ cd SOURCES
$ mkdir -p modules/example/manifests
$ cat <<EOF>modules/example/manifests/init.pp
class example {
notify {"This is an example.": }
file {'/tmp/example':
mode => '0644',
owner => '0',
group => '0',
content => 'This is also an example.'
}
}
EOF
$ tar cjf example.com-puppet-1.0.tar.bz2 modules

Next, create a spec file for our rpm in rpmbuild/SPECS as shown in the following commands:

Name:           example.com-puppet
Version: 1.0
Release: 1%{?dist}
Summary: Puppet Apply for example.com

Group: System/Utilities
License: GNU
Source0: example.com-puppet-%{version}.tar.bz2
BuildRoot: %(mktemp -ud %{_tmppath}/%{name}-%{version}-%{release}-XXXXXX)

Requires: puppet
BuildArch:      noarch

%description
This package installs example.com's puppet configuration
and applies that configuration on the machine.


%prep

%setup -q -c
%install
mkdir -p $RPM_BUILD_ROOT/%{_localstatedir}/local/puppet
cp -a . $RPM_BUILD_ROOT/%{_localstatedir}/local/puppet

%clean
rm -rf %{buildroot}

%files
%defattr(-,root,root,-)
%{_localstatedir}/local/puppet

%post
# run puppet apply
/bin/env puppet apply --logdest syslog --modulepath=%{_localstatedir}/local/puppet/modules %{_localstatedir}/local/puppet/manifests/site.pp 

%changelog
* Fri Dec 6 2013 Thomas Uphill <thomas@narrabilis.com> - 1.0-1
- initial build

Then use rpmbuild to build the rpm based on this spec, as shown in the following command:

$ rpmbuild -ba example.com-puppet.spec
…
Wrote: /home/builder/rpmbuild/SRPMS/example.com-puppet-1.0-1.el6.src.rpm
Wrote: /home/builder/rpmbuild/RPMS/noarch/example.com-puppet-1.0-1.el6.noarch.rpm

Now, deploy a node and copy the rpm onto that node. Verify that the node installs Puppet and then does a Puppet apply run.

# yum install example.com-puppet-1.0-1.el6.noarch.rpm 
Loaded plugins: downloadonly
…
Installed:
  example.com-puppet.noarch 0:1.0-1.el6 
Dependency Installed:
  augeas-libs.x86_64 0:1.0.0-5.el6
...
  puppet-3.3.2-1.el6.noarch
…
Complete!

Verify that the file we specified in our package has been created by using the following command:

# cat /tmp/example
This is also an example.

Now, if we are going to rely on this system of pushing Puppet to nodes, we have to make sure we can update the rpm on the clients and we have to ensure that the nodes still run Puppet regularly so as to avoid configuration drift (the whole point of Puppet). There are many ways to accomplish these two tasks. We can put the cron definition into the post section of our rpm:

%post
# install cron job
/bin/env puppet resource cron 'example.com-puppet' command='/bin/env puppet apply --logdest syslog --modulepath=%{_localstatedir}/local/puppet/modules %{_localstatedir}/local/puppet/manifests/site.pp' minute='*/30' ensure='present'

We could have a cron job be part of our site.pp, as shown in the following command:

cron { 'example.com-puppet':
  ensure      => 'present',
  command => '/bin/env puppet apply --logdest syslog --modulepath=/var/local/puppet/modules /var/local/puppet/manifests/site.pp',
  minute  => ['*/30'],
  target   => 'root',
  user  => 'root',
}

To ensure the nodes have the latest version of the code, we can define our package in the site.pp.

package {'example.com-puppet':  ensure => 'latest' }

In order for that to work as expected, we need to have a yum repository for the package and have the nodes looking at that repository for packages.

Creating a YUM repository is a very straightforward task. Install the createrepo rpm and then run createrepo on each directory you wish to make into a repository.

# mkdir /var/www/html/puppet
# yum install createrepo
…
Installed:
 createrepo.noarch 0:0.9.9-18.el6   
# chown builder /var/www/html/puppet
# sudo -iu builder
$ mkdir /var/www/html/puppet/{noarch,SRPMS}
$ cp /home/builder/rpmbuild/RPMS/noarch/example.com-puppet-1.0-1.el6.noarch.rpm /var/www/html/puppet/noarch
$ cp rpmbuild/SRPMS/example.com-puppet-1.0-1.el6.src.rpm /var/www/html/puppet/SRPMS
$ cd /var/www/html/puppet
$ createrepo noarch
$ createrepo SRPMS

Our repository is ready, but we need to export it with the web server to make it available to our nodes. This rpm contains all our Puppet code, so we need to ensure that only the clients we wish get access to the files. We'll create a simple listener on port 80 for our Puppet repository

Listen 80
<VirtualHost *:80>
  DocumentRoot /var/www/html/puppet
</VirtualHost>

Now, the nodes need to have the repository defined on them so they can download the updates when they are made available via the repository. The idea here is that we push the rpm to the nodes and have them install the rpm. Once the rpm is installed, the yum repository pointing to updates is defined and the nodes continue updating themselves.

yumrepo { 'example.com-puppet':
  baseurl  => 'http://puppet.example.com/noarch',
  descr    => 'example.com Puppet Code Repository',
  enabled  => '1',
  gpgcheck => '0',
}

So to ensure that our nodes operate properly, we have to make sure of the following things:

A default node in our masterless configuration requires that the cron task and the repository be defined. If you wish to segregate your nodes into different production zones (such as development, production, and sandbox), I would use a repository management system like Pulp. Pulp allows you to define repositories based on other repositories and keeps all your repositories consistent.