Jenkins Continuous Integration Cookbook - Second Edition

Over 90 recipes to produce great results from Jenkins using pro-level practices, techniques, and solutions

Jenkins Continuous Integration Cookbook - Second Edition

This ebook is included in a Mapt subscription
Alan Mark Berg

Over 90 recipes to produce great results from Jenkins using pro-level practices, techniques, and solutions
$0.00
$15.00
$49.99
$29.99p/m after trial
RRP $29.99
RRP $49.99
Subscription
eBook
Print + eBook
Start 30 Day Trial
Subscribe and access every Packt eBook & Video.
 
  • 4,000+ eBooks & Videos
  • 40+ New titles a month
  • 1 Free eBook/Video to keep every month
Start Free Trial
 
Preview in Mapt

Book Details

ISBN 139781784390082
Paperback408 pages

Book Description

Jenkins is an award-wining and one of the most popular Continuous Integration servers in the market today. It was designed to maintain, secure, communicate, test, build, and improve the software development process.

This book starts by examining the most common maintenance tasks. This is followed by steps that enable you to enhance the overall security of Jenkins. You will then explore the relationship between Jenkins builds and Maven pom.xml. Then, you will learn how to use plugins to display code metrics and fail builds to improve quality, followed by how to run performance and functional tests against a web application and web services. Finally, you will see what the available plugins are, concluding with best practices to improve quality.

 

 

Read an Extract from the book

Analyzing project data with the R plugin

This recipe describes how to use R to process metrics on each file in your project workspace. The recipe does this by traversing the workspace and collecting a list of files of a particular extension such as Java. The R script then analyzes each file individually and finally plots the results in a graphical format to a PDF file. The workflow is common to almost all quality-related analysis of software projects. This recipe is easily customized for tasks that are more complex.

In this example, we are looking at the size in words of the text files, printing to the console the names of large files and plotting the sizes of all files. From the visual representation, you can easily see which files are particularly large. If your property file is much larger than the other property files, it is probably corrupt. If a Java file is too large, it is difficult to read and understand.

How to do it...

  1. Create a free-style job with the name ch5.R.project.data.
  2. In the Source Code Management section, select Subversion.
  3. Add the Repository URL as https://source.sakaiproject.org/svn/profile2/trunk.
  4. In the build section, under Add build step, select Execute R script.
  5. In the Script text area add the following code:
    processFile
    <- function(file){
      text <- readLines(file,encoding="UTF-8")
      if (length(text)> 500) print(file)
      length(text)
    }
    javaFiles <-
    list.files(Sys.getenv('WORKSPACE'), recursive 
    
    = TRUE, full.names = TRUE, pattern = "\\.java$")
    propertiesFiles
    <- list.files(Sys.getenv('WORKSPACE'), 
    
    recursive = TRUE, full.names = TRUE, pattern = "\\.properties$")
    resultJava <-
    sapply(javaFiles, processFile)
    resultProperties
    <- sapply(propertiesFiles,processFile)
    warnings()
    filename
    <-paste('Lengths_JAVA_',Sys.getenv('BUILD_NUMBER'),'.pdf',sep ="")
    pdf(file=filename)
    hist(resultJava,main="Frequency
    of length of JAVA files")
     
    filename <-
    
    paste('Lengths_Properties_',Sys.getenv('BUILD_NUMBER'),'.pd
    
    f',sep="")
    pdf(file=filename)
    hist(resultProperties,main="Frequency
    of length of Property files")
  6. Click on the Save button.
  7. Click on the Build Now icon.
  8. Review the console output from the build. It should appear similar to the following:
    At revision 313948
    no change for 
     https://source.sakaiproject.org/svn/profile2/trunk since 
     the previous build
    [ch5.R.project.data] $ Rscript /tmp/hudson7641363251840585368.R
    [1] "/var/lib/jenkins/workspace/ch5.project.data/api/src/java/o
     rg/sakaiproject/profile2/logic/SakaiProxy.java"
    [1] 
     "/var/lib/jenkins/workspace/ch5.project.data/impl/src/java/org/sakaiproject/profile2/conversion/ProfileConverter.java"
    [1] 
     "/var/lib/jenkins/workspace/ch5.project.data/impl/src/java/
     org/sakaiproject/profile2/dao/impl/ProfileDaoImpl.java"
    14: In readLines(file, encoding = "UTF-8") :
      incomplete final line found on 
     '/var/lib/jenkins/workspace/ch5.project.data/tool/src/java/
     org/apache/wicket/markup/html/form/upload/MultiFileUploadFi
     eld_ca_ES.properties'
    Finished: SUCCESS
  9. Visit the workspace and review the files Lengths_Properties_1.pdf, Lengths_JAVA_1.pdf.
  10. Notice the straggler files with a large number of lines. Property files should be of roughly similar length, as they contain, in this case, the international translations for the GUI.

 This feels like a well-balanced project, as there are only a few files that have a large number of lines of code.

How it works...

You loaded in the profile2 tool from subversion https://source.sakaiproject.org/svn/profile2/trunk. This code is used by millions of students around the world and represents mature, realistic production code.

Within your R script, you defined a function that takes a filename as input and then reads the file into a text object. The function then checks to see whether the number of lines is greater than 500. If it is greater than 500 lines then the filename is printed to the console output. Finally, the function returns the number of lines in the text file.

processFile <- function(file){
  text <- readLines(file,encoding="UTF-8")
  if (length(text)> 500) print(file)
  length(text)
}

Next, the script discovers the property and Java files under the workspace. The file search is filtered by the value defined in the pattern argument. In this case, .java:

javaFiles <- list.files(Sys.getenv('WORKSPACE'), recursive = TRUE, 
 full.names = TRUE, pattern = "\\.java$")
propertiesFiles <- list.files(Sys.getenv('WORKSPACE'), recursive = 
 TRUE, full.names = TRUE, pattern = "\\.properties$")

The list of filenames is passed one name at a time to the processFile function you have previously defined. The results are a list of file lengths that are stored in the resultJava and resultProperties objects:

resultJava <- sapply(javaFiles, processFile)
resultProperties <- sapply(propertiesFiles,processFile)

The warnings() function produces a list of issues generated while running the sapply command:

14: In readLines(file, encoding = "UTF-8") :
  incomplete final line found on 
 '/var/lib/jenkins/workspace/ch5.project.data/tool/src/java/org/apa
 che/wicket/markup/html/form/upload/MultiFileUploadField_ca_ES.prop
 erties'

This is stating that a new line was expected at the end of the file. It is not a critical issue. Showing the warnings is a helpful approach to discovering corrupted files.

Finally, we generate two histograms of the results, one for the Java file and the other for the properties files. The filename is created from a constant string followed by the BUILD_NUMBER environment variable that is set uniquely for each build. The pdf function tells R that the output is to be stored in a PDF file and the hist function draws a histogram of the results:

filename <-paste('Lengths_JAVA_',Sys.getenv('BUILD_NUMBER'),'.pdf',sep="")
pdf(file=filename)
hist(resultJava,main="Frequency of length of JAVA files")

There's more...

When writing R code for processing your files, don't reinvent the wheel. R has many libraries for manipulating text. The stringi library is one example (http://cran.r-project.org/web/packages/stringi/stringi.pdf). Here is some example code that counts the number of words in a text file:

library(stringi)
processFile <-function(file){
stri_stats_latex(readLines(file,encoding="UTF-8"))
}
results<-processFile(file.choose())
paste("Number of words in file:", results[4])

The script defines the function processFile. The function requires a filename. The file is read into the stri_stats_latex function. This function is included in the stringi library. It returns a summary of the file as a vector (a series of numbers).

The file.choose() function pops up a dialog that allows you to browse your file system and choose a file. The call returns the fully qualified path to the file. It passes the value to the processFile function call. The results are stored in the results vector. The script then prints out the fourth number that is the number of words in the file.

Another interesting R package for text mining is tm: (http://cran.r-project.org/web/packages/tm/tm.pdf). The tm package has the ability to load a set of text files and analyze them in many different ways.

 

Table of Contents

Chapter 1: Maintaining Jenkins
Introduction
Using a test Jenkins instance
Backing up and restoring
Modifying the Jenkins configuration from the command line
Installing Nginx
Configuring Nginx as a reverse proxy
Reporting overall storage use
Deliberately failing builds through log parsing
Adding a job to warn of storage use violations through log parsing
Keeping in contact with Jenkins through Firefox
Monitoring via JavaMelody
Keeping track of script glue
Scripting the Jenkins CLI
Global modifications of jobs with Groovy
Signaling the need to archive
Chapter 2: Enhancing Security
Introduction
Testing for OWASP's top 10 security issues
Finding 500 errors and XSS attacks in Jenkins through fuzzing
Improving security via small configuration changes
Avoiding sign-up bots with JCaptcha
Looking at the Jenkins user through Groovy
Working with the Audit Trail plugin
Installing OpenLDAP
Using Script Realm authentication for provisioning
Reviewing project-based matrix tactics via a custom group script
Administering OpenLDAP
Configuring the LDAP plugin
Installing a CAS server
Enabling SSO in Jenkins
Exploring the OWASP Dependency-Check plugin
Chapter 3: Building Software
Introduction
Plotting alternative code metrics in Jenkins
Running Groovy scripts through Maven
Manipulating environmental variables
Running Ant through Groovy in Maven
Failing Jenkins jobs based on JSP syntax errors
Configuring Jetty for integration tests
Looking at license violations with Rat
Reviewing license violations from within Maven
Exposing information through build descriptions
Reacting to generated data with the groovy-postbuild plugin
Remotely triggering jobs through the Jenkins API
Adaptive site generation
Chapter 4: Communicating Through Jenkins
Introduction
Skinning Jenkins with the simple themes plugin
Skinning and provisioning Jenkins using a WAR overlay
Generating a home page
Creating HTML reports
Efficient use of views
Saving screen space with the Dashboard View plugin
Making noise with HTML5 browsers
An extreme view for reception areas
Mobile presentation using Google Calendar
Mobile apps for Android and iOS
Knowing your audience with Google Analytics
Simplifying powerful visualizations using the R plugin
Chapter 5: Using Metrics to Improve Quality
Introduction
Estimating the value of your project through sloccount
Looking for "smelly" code through code coverage
Activating more PMD rulesets
Creating custom PMD rules
Finding bugs with FindBugs
Enabling extra FindBug rules
Finding security defects with FindBugs
Verifying HTML validity
Reporting with JavaNCSS
Checking code style using an external pom.xml file
Faking Checkstyle results
Integrating Jenkins with SonarQube
Analyzing project data with the R plugin
Chapter 6: Testing Remotely
Introduction
Deploying a WAR file from Jenkins to Tomcat
Creating multiple Jenkins nodes
Custom setup scripts for slave nodes
Testing with FitNesse
Activating FitNesse HtmlUnit fixtures
Running Selenium IDE tests
Triggering failsafe integration tests with Selenium WebDriver
Creating JMeter test plans
Reporting JMeter performance metrics
Functional testing using JMeter assertions
Enabling Sakai web services
Writing test plans with SoapUI
Reporting SoapUI test results
Chapter 7: Exploring Plugins
Introduction
Personalizing Jenkins
Testing and then promoting builds
Fun with pinning JSGames
Looking at the GUI samples plugin
Changing the help of the FileSystem SCM plugin
Adding a banner to job descriptions
Creating a RootAction plugin
Exporting data
Triggering events on startup
Groovy hook scripts and triggering events on startup
Triggering events when web content changes
Reviewing three ListView plugins
Creating my first ListView plugin

What You Will Learn

  • Integrate Jenkins with LDAP and SSO solutions
  • Maintain and secure Jenkins
  • Run an integration server firing automatic functional and performance tests
  • Communicate through social media and by plotting custom data
  • Skin Jenkins to your corporate look and feel
  • Refine the use of code metrics to improve quality
  • Write your first custom Jenkins plugin
  • Apply tweaks to optimize your use of Jenkins

Authors

Table of Contents

Chapter 1: Maintaining Jenkins
Introduction
Using a test Jenkins instance
Backing up and restoring
Modifying the Jenkins configuration from the command line
Installing Nginx
Configuring Nginx as a reverse proxy
Reporting overall storage use
Deliberately failing builds through log parsing
Adding a job to warn of storage use violations through log parsing
Keeping in contact with Jenkins through Firefox
Monitoring via JavaMelody
Keeping track of script glue
Scripting the Jenkins CLI
Global modifications of jobs with Groovy
Signaling the need to archive
Chapter 2: Enhancing Security
Introduction
Testing for OWASP's top 10 security issues
Finding 500 errors and XSS attacks in Jenkins through fuzzing
Improving security via small configuration changes
Avoiding sign-up bots with JCaptcha
Looking at the Jenkins user through Groovy
Working with the Audit Trail plugin
Installing OpenLDAP
Using Script Realm authentication for provisioning
Reviewing project-based matrix tactics via a custom group script
Administering OpenLDAP
Configuring the LDAP plugin
Installing a CAS server
Enabling SSO in Jenkins
Exploring the OWASP Dependency-Check plugin
Chapter 3: Building Software
Introduction
Plotting alternative code metrics in Jenkins
Running Groovy scripts through Maven
Manipulating environmental variables
Running Ant through Groovy in Maven
Failing Jenkins jobs based on JSP syntax errors
Configuring Jetty for integration tests
Looking at license violations with Rat
Reviewing license violations from within Maven
Exposing information through build descriptions
Reacting to generated data with the groovy-postbuild plugin
Remotely triggering jobs through the Jenkins API
Adaptive site generation
Chapter 4: Communicating Through Jenkins
Introduction
Skinning Jenkins with the simple themes plugin
Skinning and provisioning Jenkins using a WAR overlay
Generating a home page
Creating HTML reports
Efficient use of views
Saving screen space with the Dashboard View plugin
Making noise with HTML5 browsers
An extreme view for reception areas
Mobile presentation using Google Calendar
Mobile apps for Android and iOS
Knowing your audience with Google Analytics
Simplifying powerful visualizations using the R plugin
Chapter 5: Using Metrics to Improve Quality
Introduction
Estimating the value of your project through sloccount
Looking for "smelly" code through code coverage
Activating more PMD rulesets
Creating custom PMD rules
Finding bugs with FindBugs
Enabling extra FindBug rules
Finding security defects with FindBugs
Verifying HTML validity
Reporting with JavaNCSS
Checking code style using an external pom.xml file
Faking Checkstyle results
Integrating Jenkins with SonarQube
Analyzing project data with the R plugin
Chapter 6: Testing Remotely
Introduction
Deploying a WAR file from Jenkins to Tomcat
Creating multiple Jenkins nodes
Custom setup scripts for slave nodes
Testing with FitNesse
Activating FitNesse HtmlUnit fixtures
Running Selenium IDE tests
Triggering failsafe integration tests with Selenium WebDriver
Creating JMeter test plans
Reporting JMeter performance metrics
Functional testing using JMeter assertions
Enabling Sakai web services
Writing test plans with SoapUI
Reporting SoapUI test results
Chapter 7: Exploring Plugins
Introduction
Personalizing Jenkins
Testing and then promoting builds
Fun with pinning JSGames
Looking at the GUI samples plugin
Changing the help of the FileSystem SCM plugin
Adding a banner to job descriptions
Creating a RootAction plugin
Exporting data
Triggering events on startup
Groovy hook scripts and triggering events on startup
Triggering events when web content changes
Reviewing three ListView plugins
Creating my first ListView plugin

Book Details

ISBN 139781784390082
Paperback408 pages
Read More

Read More Reviews