Mastering Jenkins: Configure and extend Jenkins to architect, build, and automate efficient software delivery pipelines

Jenkins, as you are probably aware, was built using Java. It's cross platform and supports a wide variety of engineering patterns and technologies. It can effectively support organizations of all sizes. Jenkins offers endless automation possibilities and even has a vibrant plugin ecosystem.

In this section, we will cover configuration techniques aimed at advancing our knowledge of the Jenkins architecture. Together, we will lay a foundation for reliable automation and future scalability by learning how to configure the Jenkins main operating environment.

Jenkins offers a cross-platform user experience by running on a Java Virtual Machine (JVM). A JVM is a powerful virtualization technology used to execute Java bytecode instructions, and provides a consistent software platform regardless of the host OS.

The Jenkins community has created native installation packages for Microsoft Windows, Mac OS X, Linux, and Unix. If Jenkins is not already installed (this book assumes it is), it's highly recommended that you use one of the community-provided packages if possible. If a native package is not available, Jenkins an most likely still be configured to run, providing that the host operating system supports any of the following web architectures:

The initial installation of Jenkins on Microsoft Windows is straightforward. The Microsoft Windows Jenkins MSI package comes complete with the Java Runtime Environment (JRE) prerequisite and Microsoft .NET 2.0 framework. This bundling provides a seamless Jenkins installation experience, and alleviates the need for any external prerequisite software installations.

By using the MSI installation package, the Jenkins installation wizards will automatically install itself as a Windows service. By installing Jenkins as a Microsoft Windows service it will become easier to manage and control. Windows services provide an easy way to specify what the system should do when Jenkins crashes.

Once Jenkins has been installed we will want to verify that Jenkins has properly been installed and configured as a Windows service. To do this we will need to open the Services area in the Control Panel and locate the entry for Jenkins. To navigate to the Windows services management area, go to Start-> Control Panel -> Administrative Tools -> Services.

If Jenkins is installed properly as a Windows service, we will see an entry for it listed in the Windows Services panel, as shown in the following screenshot:

Figure 1-1: The Windows Services panel

The Windows service properties dialog displays the available options for Windows services. From this screen, we can modify how the service is started, what user account the service runs under, and even manage how Microsoft Windows recovers the Jenkins service if it encounters any problems.

To open the Windows service properties panel for Jenkins, double-click on the Jenkins entry on the main Windows Services panel. Windows will then display the Jenkins Properties dialog, as shown here:

Figure 1-2: The Jenkins Properties dialog

Let's make Jenkins more fault tolerant. To do this, we need to modify the Jenkins Windows service definition and implement a more reliable way for our Jenkins service to recover from failure. Click on the Recovery tab, as shown here:

Figure 1-3: The Recovery tab in the Jenkins Properties dialog

From here, we can direct Microsoft Windows to automatically restart the Jenkins service if it fails. In the example shown in the preceding screenshot, we have implemented fault tolerance for Jenkins by updating the fields in the dialog.

To proceed update your Jenkins service to reflect the above implementation. Click on Apply and restart the Jenkins service.

<arguments>-Xrs –Xmx512m -Dhudson.lifecycle=hudson.lifecycle.WindowsServiceLifecycle -jar "%BASE%\jenkins.war" --httpPort=8080</arguments>

Jenkins offers a wide spectrum of support for the Linux and Unix operating systems. Its cross-platform capabilities have made it a very popular automation tool. The Jenkins community has created native installation packages for most Linux and Unix distributions. Currently, there are installation packages available for the following Linux and Unix flavors:

The easiest way to install Jenkins on a Linux or Unix system is to use a standard package manager, such as YUM, OpenCSW, IPS, or Aptitude. For the purpose of brevity, we will focus primarily on CentOS (YUM) and Debian oriented (APT) distributions.

JAVA_ARGS=-Xmx=512m

One of the newer web server solutions to take the Internet by storm is NGinX (pronounced Engine X). Developed in 2004 under the supervision of Igor Sysoev, NGINX was created to facilitate scalability and the load-balancing requirements of high-traffic web sites. Since its inception, this tool has gained wide acceptance and notoriety. Let's look at how to apply a reverse proxy to Jenkins. This can be accomplished in a straightforward manner. Let's take a few minutes to look at how to achieve this.

If NGINX has not already been installed on the target system, the first step will be to install it. NGINX can easily be installed onto an Ubuntu/Debian or CentOS-based system using the following terminal commands:

CENTOS#> yum install nginx
DEBIAN#> apt-get install nginx

Upon completing the installation of the NGINX web server, verify that the installation was successful by executing the nginx –v command. This will display the version information for the NGINX web server on the terminal and provide us with the assurance that it is installed properly.

Now that the NGINX web server has been installed onto the target system, the system will need to be configured to act as a reverse proxy for the Jenkins JVM. To accomplish this, simply update the nginx configuration files to contain a proxy pass. The configuration files for nginx on Ubuntu can be found in the following location:

/etc/nginx/sites-enabled/default

An example (provided at http://www.jenkins-ci.org) of a Jenkins proxy pass entry with Jenkins running under a subdomain (Jenkins.domain.com) is provided below.

server {
    listen 80;
    server_name jenkins.domain.com;
    return 301 https://$host$request_uri;
}

server {

    listen 80;
    server_name jenkins.domain.com;

    location / {

      proxy_set_header        Host $host;
      proxy_set_header        X-Real-IP $remote_addr;
      proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header        X-Forwarded-Proto $scheme;

      # Fix the "It appears that your reverse proxy set up is broken" error.
      proxy_pass          http://127.0.0.1:8080;
      proxy_read_timeout  90;

      proxy_redirect      http://127.0.0.1:8080 https://jenkins.domain.com;
    }
  }

Once the configuration file has been updated, save the file to the disk, and restart nginx with the following command:

#>sudo service nginx restart

For the Jenkins UI and the NGINX reverse proxy to properly integrate, the context paths of Jenkins and the NGINX subdirectory must match. For example, if the Jenkins system is configured to use a context path of http://localhost:8080/jenkins, the proxy pass context defined in the web server's configuration file must also reflect the /Jenkins suffix.

To set the context path for Jenkins, add the --prefix= entry to the JENKINS_ARGS= property. An example of this configuration entry is provided below.

--prefix=/Jenkins
--prefix=/somecontextpathhere

The JENKINS_ARGS configuration line is located inside the Jenkins startup bash/dash script. This file is typically found in one of the following locations on the filesystem (dependent on the Linux distribution):

/etc/default/Jenkins

/etc/sysconfig/Jenkins (line 151)

Once everything has been configured, restart the NGINX and Jenkins services to finalize the implementation of the reverse proxy redirect solution.

After this has been completed, navigate from a web browser to your Jenkins URL on port 80, and verify that the Jenkins UI behaves properly.

The Apache HTTP Server Project (referred to as Apache) was first released to the public in April 1995. Apache represents a cornerstone of the Internet and is highly acclaimed. Since its initial release, Apache has become one of the most widely adopted web server platforms around the world. Apache thrives to this day, and has a vast array of modules that extend its functionality.

Running Jenkins on a nonstandard web port is a limitation that you may wish to address. It requires users to specify the port as part of the URL, which can become a hassle to manage. To address this concern, it's useful to run Jenkins behind an Apache proxy. Apache provides an extensive set of features that can benefit Jenkins users. Some benefits of this solution include:

On Linux and Unix hosts, all TCP and UDP ports lower than 1024 are considered privileged. A privileged port is one where the services exposed are running under a privileged user account (typically, root). Running Jenkins as root is considered dangerous and is, therefore, not recommended. The prescribed user account under which the Jenkins service executes is jenkins.

The privileged port restriction can create a bit of a problem for Jenkins administrators. We may want Jenkins to be accessible on standard web port 80, but still want to run it under the Jenkins user account. By running Jenkins behind Apache, we can address this issue while still adhering to best practices.

One possible solution is to run Jenkins behind Apache and use mod_proxy to internally redirect traffic. This solution will allow us to get Jenkins to respond to requests on a privileged port without executing it as the root user, or changing its port configuration.

Apache's mod_proxy operates by forwarding incoming TCP/UDP port 80 requests to Jenkins on port 8080. Any responses from Jenkins are forwarded back to port 80. This creates a seamless experience for web users, while allowing Jenkins to remain on its original port.

To configure Jenkins to use Apache's mod_proxy module, we will need a basic installation of the Apache web server (http://www.apache.org). If Apache is not already installed on your system, you can use the standard package manager for your Linux or Unix distribution to install it. Here are some example terminal commands to install Apache on Linux:

Once Apache is installed on the host, we will need to ensure that mod_proxy is installed and loaded also.

On RHEL-/CentOS-based systems, mod_proxy is typically installed at the same time as Apache. For Debian-based systems, you may need to install the Apache mod_proxy module specifically. You can look at your Apache modules folder on your system to see whether the mod_proxy.so and mod_proxy_http.so files are present.

To ensure mod_proxy is operating properly we need to ensure that Apache loads the mod_proxy.so and mod_proxy_http.so modules when Apache initializes. Apache's configuration file is usually httpd.conf or apache2.conf, depending on your system. Let's open up the Apache configuration file in an editor (vi, nano, and so on) and ensure that we have the proper LoadModule directives.

Next we will want to direct Apache to load our mod_proxy modules. The proper Apache configuration lines are provided below.

LoadModule proxy_module modules/mod_proxy.so
LoadModule proxy_http_module modules/mod_proxy_http.so

Once the LoadModule changes to the Apache configuration file have been completed, we will need to configure the port 80 VirtualHost XML block and create a reverse proxy. An example of an Apache virtual host entry for a Jenkins instance and proxy running on port 80 is provided below.

<VirtualHost *:80>
ServerAdmin  webmaster@localhost
ProxyRequests     Off
ProxyPreserveHost On
AllowEncodedSlashes On

    <Proxy *>
        Order deny,allow
        Allow from all
    </Proxy>

    ProxyPass         /  http://localhost:8080/ nocanon
    ProxyPassReverse  /  http://localhost:8080/</VirtualHost>

Now that the virtual host configuration entries have been added, save the configuration file to disk and restart Apache. Depending on the distribution, the commands necessary to restart Apache may vary slightly. Examples for CentOS and Debian are provided here:

For the Jenkins UI and the Apache reverse proxy to properly integrate, the context paths of Jenkins and the Apache subdirectory must match. For example, if the Jenkins system is configured to utilize a context path of http://localhost:8080/jenkins, the proxy pass context defined in the Apache configuration file must also reflect the /Jenkins suffix.

To set the context path for Jenkins, add the --prefix= entry to the JENKINS_ARGS= property. An example of this configuration entry is provided below.

--prefix=/Jenkins
--prefix=/somecontextpathhere

The JENKINS_ARGS configuration line is located inside the Jenkins startup bash/dash script. This file is typically found in one of the following locations on the filesystem that houses Jenkins (depending on your Linux distribution):

/etc/default/Jenkins

/etc/sysconfig/Jenkins (line 151)

Once everything has been configured, restart the Apache one more time as well as the Jenkins service to finalize the implementation of the reverse proxy solution. To verify everything is functioning properly navigate from a web browser to your Jenkins URL on port 80 and verify that the Jenkins UI behaves as expected.

Operating a Jenkins master from a single machine inherently creates a Single Point of Failure (SPOF). If the Jenkins master was lost for any reason, rebuilding it could be time consuming and may not be very easy. If the Jenkins master is lost or destroyed, there may be a crippling impact on your organization's ability to build, test, or release. Let's address this and create a disaster-recovery plan for Jenkins to ensure a high level of availability and quick turnaround time for any failures that may occur.

Jenkins snapshot backups

Snapshot backups of the $JENKINS_HOME directory can provide a level of fault tolerance, and offer an effective solution for system-wide backups. There are a number of Jenkins plugins that can effectively assist us in creating snapshot backups. The most popular backup plugins include BackUp, thinBackup, and SCM Sync configuration. You will need to choose the one that best suits your specific needs.

All of the above listed plugins are available from the Jenkins Manage Plugins screen. You can access this area by opening Jenkins, logging in as an administrator, and then going to Jenkins -> Manage Jenkins -> Manage Plugins -> Available plugins.

The following screenshot shows some of the available backup plugins on the Jenkins Manage Plugins screen:

Figure 1-4: The Manage Jenkins screen

Installing any of the available backup plugins (or any plugin for that matter) can be accomplished by simply marking the appropriate checkbox and clicking on the Download now and install after restart button.

Once the plugin has been installed, it is highly recommended that you read any usage instructions and documentation. A direct link to the documentation for all Jenkins plugins can be found at https://wiki.jenkins-ci.org/display/JENKINS/Plugins.

Tip

Downloading the example code

You can download the example code files for all Packt books you have purchased from your account at http://www.packtpub.com. If you purchased this book elsewhere, you can visit http://www.packtpub.com/support and register to have the files e-mailed directly to you.

$> sudo service jenkins stop

$> which rsync
$> rsync --version

$> sudo su - Jenkins
$> sudo rsync -avuh --delete -e ssh root@jenkinsprimary:/var/lib/jenkins/* /var/lib/jenkins

$> sudo service jenkins start

$Jenkins-Primary>sudo su – root
$Jenkins-Primary>ssh-keygen –t rsa
$Jenkins-Primary>scp ~/.ssh/id_rsa.pub root@jenkins-mirror:/tmp/
$Jenkins-Mirror>sudo su – root
$Jenkins-Mirror>cat /tmp/id_rsa.pub >> ~/.ssh/authorized_keys

$Jenkins-Mirror>sudo su – root
$Jenkins-Mirror>ssh-keygen –t rsa
$Jenkins-Mirror>scp ~/.ssh/id_rsa.pub root@jenkins-primary:/tmp/
$Jenkins-Primary>sudo su – root
$Jenkins-Primary>cat /tmp/id_rsa.pub >> ~/.ssh/authorized_keys

$Jenkins-Mirror>ssh root@jenkins-primary
$Jenkins-Primary>ssh root@jenkins-mirror

*/30 * * * * /usr/bin/rsync -avuh --delete -e ssh root@jenkinsprimary:/var/lib/jenkins/* /var/lib/jenkins

For Mac OS X users, the Jenkins community provides a native PKG installer. This installer has a similar guided installation wizard as the Microsoft Windows one, and allows us to specify the traditional installation details. For users who wish to alter the Jenkins listening port or memory options, some command-line magic will be required.

To alter the Jenkins listening port, we need to explicitly define the port in the following properties file:

/Applications/Jenkins/winstone.properties

To accomplish this we will need to create the winstone.properties file if it does not already exist on our host, and detail the httpPort parameter inside it. An example of how to set httpPort in the winstone.properties file is shown here:

httpPort=80

Once created, the winstone.properties file will automatically load and override the default Jenkins port with the one specified within.

The winstone.properties file is not limited to simply altering the Jenkins listening port. There are other options available for customization as well. These options include logfile, httpListenAddress, and more. To obtain a complete list of the available override options, you can run the following commands from your OS X terminal:

cd /Applications/Jenkins java –jar Jenkins.war --help

The JVM runtime memory settings (Heap memory, PermGen, and so on) are stored in a standard properties file format (org.jenkins-ci.plist). The launch daemon retrieves the values stored in this properties file. If no such file exists, the system will use the built-in defaults. On Mac OS X, this plist file will typically reside in the following location:

/Library/Preferences/org.jenkins-ci.plist

Adjusting the Java memory options for Jenkins involves modifying the appropriate property entries inside the org.jenkins-ci.plist file. To modify this file, we can use the OS X defaults command. This command allows us to read and write entries in the plist file without fear of corruption or improper formatting. A few example descriptions and use cases for this command are detailed in the sections below.

To view all settings in the plist file, execute the following command in the command-line terminal:

sudo defaults read /Library/Preferences/org.jenkins-ci

The output of the preceding command will look like this:

{
    heapSize = 512m;
    minHeapSize = 256m;
    minPermGen = 256m;
    permGen = 512m;
    tmpdir = "/Users/Shared/Jenkins/tmp";
}

To retrieve the value of the heapSize setting from the plist file, execute the following command in the command-line terminal:

sudo defaults read /Library/Preferences/org.jenkins-ci heapSize

The output of the preceeding command will look like this:

512m

To set the value of the heapSize setting in the plist file, execute following command in the command-line terminal:

sudo defaults write /Library/Preferences/org.jenkins-ci heapSize 1024m

If any alterations are made to the org.jenkins-ci.plist file, make sure you restart Jenkins for them to take effect. To restart Jenkins from the OS X command line terminal enter the following commands in the terminal:

sudo launchctl unload /Library/LaunchDaemons/org.jenkins-ci.plist
sudo launchctl load /Library/LaunchDaemons/org.jenkins-ci.plist

<?xml version='1.0' encoding='UTF-8'?>
<sites>
  <site>
    <id>default</id>
    <url>http://updates.jenkins-ci.org/stable/update-center.json</url>
  </site>
</sites>

$JENKINS_HOME/config.xml

<?xml version='1.0' encoding='UTF-8'?>
<hudson>
  <disabledAdministrativeMonitors/>
  <version>1.0</version>
  <numExecutors>2</numExecutors>
  <mode>NORMAL</mode>
  <useSecurity>true</useSecurity>
  <authorizationStrategy class="hudson.security.AuthorizationStrategy$Unsecured"/>
  <securityRealm class="hudson.security.SecurityRealm$None"/>
  <disableRememberMe>false</disableRememberMe>
  <projectNamingStrategy class="jenkins.model.ProjectNamingStrategy$DefaultProjectNamingStrategy"/>
  <workspaceDir>${ITEM_ROOTDIR}/workspace</workspaceDir>
  <buildsDir>${ITEM_ROOTDIR}/builds</buildsDir>
  <markupFormatter class="hudson.markup.EscapedMarkupFormatter"/>
  <jdks/>
  <viewsTabBar class="hudson.views.DefaultViewsTabBar"/>
  <myViewsTabBar class="hudson.views.DefaultMyViewsTabBar"/>
  <clouds/>
  <slaves>
    <slave>
      <name>Windows 2012</name>
      <description></description>
      <remoteFS></remoteFS>
      <numExecutors>1</numExecutors>
      <mode>NORMAL</mode>
      <retentionStrategy class="hudson.slaves.RetentionStrategy$Always"/>
      <launcher class="hudson.plugins.sshslaves.SSHLauncher" plugin="ssh-slaves@1.9">
        <host></host>
        <port>22</port>
        <credentialsId>0bb868e0-2cd6-4ab2-9781-a373d914cb85</credentialsId>
        <maxNumRetries>0</maxNumRetries>
        <retryWaitTime>0</retryWaitTime>
      </launcher>
      <label>Windows Build Pool</label>
      <nodeProperties/>
      <userId>anonymous</userId>
    </slave>
  </slaves>
  <scmCheckoutRetryCount>0</scmCheckoutRetryCount>
  <views>
    <hudson.model.AllView>
      <owner class="hudson" reference="../../.."/>

    <name>All</name>
      <filterExecutors>false</filterExecutors>
      <filterQueue>false</filterQueue>
      <properties class="hudson.model.View$PropertyList"/>
    </hudson.model.AllView>
    <listView>
      <owner class="hudson" reference="../../.."/>
      <name>Build.TestApp</name>
      <filterExecutors>false</filterExecutors>
      <filterQueue>false</filterQueue>
      <properties class="hudson.model.View$PropertyList"/>
      <jobNames>
        <comparator class="hudson.util.CaseInsensitiveComparator"/>
      </jobNames>
      <jobFilters/>
      <columns>
        <hudson.views.StatusColumn/>
        <hudson.views.WeatherColumn/>
        <hudson.views.JobColumn/>
        <hudson.views.LastSuccessColumn/>
        <hudson.views.LastFailureColumn/>
        <hudson.views.LastDurationColumn/>
        <hudson.views.BuildButtonColumn/>
      </columns>
      <recurse>false</recurse>
    </listView>
  </views>
  <primaryView>All</primaryView>
  <slaveAgentPort>0</slaveAgentPort>
  <label></label>
  <nodeProperties/>
  <globalNodeProperties/>
</hudson>

<?xml version='1.0' encoding='UTF-8'?>
<project>
  <actions/>
  <description></description>
  <keepDependencies>false</keepDependencies>
  <properties/>
  <scm class="hudson.scm.NullSCM"/>
  <canRoam>true</canRoam>
  <disabled>false</disabled>
  <blockBuildWhenDownstreamBuilding>false</blockBuildWhenDownstreamBuilding>
  <blockBuildWhenUpstreamBuilding>false</blockBuildWhenUpstreamBuilding>
  <triggers/>
  <concurrentBuild>false</concurrentBuild>
  <builders/>
  <publishers/>
  <buildWrappers/>
</project>

In this chapter, we have focused on techniques that taught us how to create and manage a Jenkins instance. This represents the foundation for automation. By mastering these configuration solutions, you have learned ways in which you can make your Jenkins system stable and scalable. In this chapter, we have discussed port changes, memory management, backups, LTS release migration, RSYNC mirroring, and configuration data.

From here, we will move on to discover the power of Jenkins slave agents, complete job management, UI administration, and much more. As we progress through our journey, we will delve into the exciting features Jenkins provides, and begin leveraging Jenkins to automate additional aspects of the SDLC.