Knowing when your build breaks
First, let us define a continuous integration, or CI server. A CI server is an application that is used to monitor your project's build and ensure the health of your source repository.
In a software development setup, a project includes a number of developers. If it is a big project, expect the team to be larger as well. Apart from this, a team can be physically distributed in different locations. An open source project is a good example of this. Developers in open source communities are often located in different parts of the world. All these developers commit or check their code into a central source repository. Sometimes though, people only tend to build the component which they have made changes to (especially when building the entire project takes a while!). Moreover, when that component builds successfully, they commit their changes without checking whether there are dependent components that had been affected by these changes. Now, what if it turned out that one dependent component's build failed because of those changes? No one would know until someone from the team updated his or her local copy and ran a full build. How can everyone else proceed if they were then unable to build the project?
If the developer who discovered the broken build were a Good Samaritan, he would try to fix the build on his own (which would probably take a while). Tracking down and identifying which changes caused the build to break is a tedious job. Either that or the developer would notify his team mates about the broken build so that whoever caused the build to fail would know and would be able to fix the build. However, what if the person who caused the build to break has just left for a vacation? His teammates could either track him down and bug him while he is on vacation, or they could fix it themselves. Either of which would eat up a lot of development time!
If only they had their project source in a CI server:
- They would know that they broke some component in the project when the build failed—whether it was a compilation failure or a test failure, because a CI server ensures that the entire source code builds and all the tests pass.
- They would know that the build is broken because the CI server would notify them when it encountered the build failure.
- They would know who caused the build to break, and they can contact that person and help them to fix it.
- They would know which changes caused the build to break as this information is available in the CI server.
- They would know when there is a problem in their build and deployment infrastructure. For example, when their source repository is down or their deployment repository is not accessible. We did not see these issues in the scenario above but they do happen.
It is obvious from our definition and list above what the importance of a CI server is. However, we will see in the following sections that it's useful for more than just that.
Setting up Continuum
Continuum is an open source continuous integration server that has support for different types of projects such as Maven 2, Maven 1, Ant, and shell projects. Some of its features include build automation, release management, source control management, and build statistics.
Archiva and Continuum used to be sub-projects of Maven. Continuum officially became a top-level project at the ASF in February 2008, just a couple of months before Archiva.
Setting up a source repository
We start off by setting up our source repository where we will be adding the example application that we have been utilizing. The source repository server that we will be using is Subversion or SVN for short. More details about Subversion can be found at http://subversion.tigris.org/. First, let's download and install Subversion as specified from their site http://subversion.tigris.org/getting.html#binary-packages. We can then set up our local SVN repository by executing the following command:
$ svnadmin create /path/to/data/svn
Make sure that you replace /path/to from the above command with the actual directory path where you will put your SVN data. This applies to all the commands with the same convention in this article.
Before we add our example application centrepoint and its parent POM effectivemaven-parent in the SVN repository, let's first create the appropriate directories in the source repository by executing the following commands:
$ svn mkdir file://localhost/path/to/data/svn/effectivemaven-parent/
$ svn mkdir file://localhost/path/to/data/svn/effectivemaven-parent/trunk
$ svn mkdir file://localhost/path/to/data/svn/effectivemaven-parent/
$ svn mkdir file://localhost/path/to/data/svn/effectivemaven-parent/tags
$ svn mkdir file://localhost/path/to/data/svn/centrepoint/
$ svn mkdir file://localhost/path/to/data/svn/centrepoint/trunk
$ svn mkdir file://localhost/path/to/data/svn/centrepoint/branches
$ svn mkdir file://localhost/path/to/data/svn/centrepoint/tags
The trunk/ directory is where the latest or current development happens. To use software lingo, this is the bleeding edge. The branches/ directory contains development code of certain versions of the project for maintenance purposes. It may also contain temporary copies of the source code in trunk/ specifically for developing major features or fixing critical bugs that may possibly break the code (they eventually get merged back to trunk/ once the changes made are stable). The tags/ directory on the other hand, contains all the released or tagged versions. A tagged version should no longer be modified as that version has already been frozen.
It is a convention to structure your SVN repository as such because it makes it easier to identify or to know where to look for specific sources. Maven and Continuum both use this structure in determining the default tag URL when releasing a project.
Now, let's import our example projects centrepoint and its parent project, effectivemaven-parent, to our SVN repository. In the sample code for this article, cd to the effectivemaven-parent first then execute the following command:
$ svn import . file://localhost/path/to/data/svn/effectivemaven-parent/
Do the same with centrepoint. cd to the centrepoint directory and execute:
$ svn import . file://localhost/path/to/data/svn/centrepoint/trunk
Before importing new projects in a source repository, make sure that there are no unnecessary files (like the Maven generated target/ directory or IDE descriptor files) present. Execute mvn clean to cleanup target/directories or mvn eclipse:clean and mvn idea:clean to remove Eclipse and IntelliJ IDEA descriptor files before adding the project in source control.
We know that Continuum supports four different types of projects: Maven 1, Maven 2, Ant, and Shell projects. In order for Continuum to build them, of course we need to have the necessary tools installed. Not all of them are required to be installed though, it really depends on the projects you will be adding to and building in Continuum. It does not make sense to have Ant installed if all of your projects are in Maven 2.
Maven 1 can be downloaded at http://maven.apache.org/maven-1.x/start/download.html.
To install Ant, we can follow the steps specified at the following web page http://ant.apache.org/manual/index.html.
Apart from the build tools above, we will also need a mail server. If you do not have a mail server installed in your machine, you can use Apache James (http://james.apache.org). Following steps 1 to 4 in http://wiki.apache.org/james/JamesQuickstart should be enough to get you started.
Installing and configuring Continuum
Now that we have everything ready, we can proceed with the installation and setup of Continuum.
Continuum can be downloaded from http://continuum.apache.org/download.html either as a standalone bundle or as a WAR file (just like Archiva). In our case, we will be using the standalone bundle.
To use the WAR file and deploy it to an application server, just follow the steps specified in http://continuum.apache.org/docs/1.3.3/installation/installation.html.
The latest version of Continuum is 1.3.3. Just download the bundle and unpack it.
I have identified below the directories and configuration files that we need to take note of in Continuum:
- conf/continuum.xml: This contains Continuum-specific configuration such as the location of the build and release output directories, working copy directory, and the base URL for accessing the webapp. The base URL is used or emails and such, which provides a URL back to the application.
- conf/jetty.xml : This is a Jetty-specific configuration. This is where the Continuum and Redback databases and mail resources are configured.
- data/: This is the default location of the Continuum and Redback databases as seen in the jetty.xml file. This is also the default base location of the working directory, build output directory, and release output directory for projects being built in Continuum.
- logs/: This is the default location of the Jetty and Continuum log files. The location of the log files can be set in the jetty-logging.xml for Jetty and apps/continuum/WEB-INF/classes/log4j.xml for Continuum-specific logs. Once we start up Continuum, we will be seeing different types of log files created in this directory. These are composed of the Continuum rolling log file, audit logs, Jetty logs and wrapper log.
We can see in the bin/ directory that there are multiple Java wrappers for starting up Continuum. The following OS platforms are supported: linux-x86-32, linuxx86-64, windows, macosx-universal-32, solaris-sparc-32, solaris-sparc-64 and solaris-x86-32.
Before we start Continuum, we must change the port it will run on. Otherwise, we will encounter an Address already in use error during startup.
To change the port, edit the configuration below in the conf/jetty.xml file. Change the default value of the jetty.port system property from 8080 to 8082.
<!-- START SNIPPET: jetty_port -->
<Set name="host"><SystemProperty name="jetty.host" /></Set>
<Set name="port"><SystemProperty name="jetty.port"
Now let's fire up Continuum. In the bin/ directory, execute ./continuum console if you want to see the logs in the console or ./continuum start to run it in the background. If you are running on Windows, you need to execute continuum.batconsole to execute it in console mode or continuum.bat install then continuum.bat start to run Continuum as a service.
I am sure you have noticed during startup that the Continuum standalone uses an embedded Jetty server similar to the one Archiva uses.
Did you know that the Archiva and Continuum standalone bundles were created using Maven plugins? These are the appassembler-mavenplugin that was used to generate the Java Service Wrappers including the wrapper configuration, and the maven-assembly-plugin, which was used to package the binaries bundle.
Once Continuum has started up successfully, we can access it in the web browser via the URL http://localhost:8082/continuum.
You can easily change Continuum's URL either by setting it in the webapp through the General Configuration page or by manually setting it in the <baseUrl> of continuum.xml. One thing to keep in mind when changing the host or the context from the default in Jetty (or in the application server), is to make sure that you change the <baseUrl> in the configuration file as well.
Accessing the above URL gives us the Create Admin page (Yes, this is just like Archiva.)
For consistency purposes, we will use the same credentials we used in Archiva for the default System Administrator user. Fill in the required fields as follows:
- Full Name: Administrator
- Email Address: firstname.lastname@example.org (this is our official phony admin email address)
- Password: admin1
After creating the admin user, we will be directed to the General Configuration page, which looks like the following:
The Working Directory is where Continuum keeps the local checkouts of the projects it is building, while the Build Output Directory and Release Output Directory are where Continuum stores the respective output of the project builds and the releases done in Continuum. We have already tackled what the Base URL is for, so we will be skipping that. As for the last two fields—Number of Builds in Parallel and Enable Distributed Builds. The contents specified in the fields above will be saved in the continuum.xml file once we hit the Save button.
As a general practice, the Working Directory, Build Output Directory and Release Output Directory should be kept separate from the Continuum installation. This is to avoid accidentally deleting these directories when upgrading Continuum.
Now, we will see how Continuum builds a project, and how it uses and works with Maven.
At a glance
Outlined below is a brief step-by-step explanation of how Continuum builds a project:
- Project build is triggered (either by the user explicitly triggering the build or through the schedule).
- Continuum adds the build to the build queue and determines the build definition and the build environment to be used for the build. You will understand what it is I am talking about once we get to the middle parts of this article.
- Continuum checks out a fresh copy or updates the project's working copy (or working directory) from the source repository. A working directory is just a checkout of the project's sources.
- Once the checkout or update is finished, Continuum builds the project using the build definition from Step 2. It actually just invokes the configured build tools such as Maven or Ant to build the project.
- When the build is finished, Continuum would collect the results of the build and stores them in the database.
- Continuum sends out notifications or messages to the project team regarding the result of the build execution—the build is broken, the build is fine, and so on.
If you later install Continuum on a server where you do not have access to the file system, you can browse the working directory through the Working Copy tab of the project in the Continuum web application.
Now enough with these concepts and let's start exercising our fingers with some CI action.
The first thing we need to do is to check out our example application and its parent POM locally. As we did not host our SVN repository to be accessible via HTTP, we will be adding our project from the filesystem. Execute:
$ svn co
$ svn co
Now we will be adding our example application to Continuum from the filesystem. In order to be able to do this, we must first enable the file:// protocol in apps/continuum/WEB-INF/classes/META-INF/plexus/application.xml. Otherwise, we will encounter a The specified resource isn't a file or the protocol used isn't allowed error message, if we attempt to add it without enabling it first.
By default, the file:// protocol for adding projects is disabled in Continuum due to security reasons. Only http, https and ftp are allowed.
Let's stop Continuum fi rst. As we are running on console, we will stop it via Ctrl+C. Use ./continuum stop if Continuum is running in the background or as a service.
Now edit apps/continuum/WEB-INF/classes/META-INF/plexus/application.xml and uncomment the file:// scheme in the following section:
<!-- <allowedScheme>file</allowedScheme> -->
Start up Continuum again and so that we can add our projects. We need to add effectivemaven-parent to Continuum first since it is our practice project's (centrepoint) parent and we will not be able to add centrepoint without it. Otherwise we will get a Missing artifact trying to build the POM. Check that its parent POM is available or add it first in Continuum error.
It is possible to add the centrepoint project to Continuum without having to add effectivemaven-parent first if and only if the effectivemaven-parent POM is already installed in the local Maven repository where Continuum is running.
Now, let's add effectivemaven-parent through the Add Maven 2.0+ Project page.
We provide the filesystem path to our check out of our example project's POM in the POM Url field, which is file:///path/to/local/checkout/of/effectivemavenparent/pom.xml. We can leave the Username and Password fields empty, as we did not set up security in our SVN repository. We will leave the rest of the fields as they are.
We can tell Continuum whether to use the default group configured in the project's POM or add it to a specific project group via the Project Group field. If we choose the default Defined by POM, a new Project Group will be created based on the <name> element set in the project's POM. The project (and its submodules) will be added under that group. Now if we choose a specific Project Group, then that's where our new project will be added.
Now, click Add and we will get a Missing "scm'' element in the POM, project Apache Maven 2: Effective Implementations Book error. This is Continuum telling us that we don't have the required information configured in our POM. Therefore, here is what we will do:
In our local checkout of the project, we add the
Continuum requires the
Then we commit our changes to the source repository.
effectivemaven-parent$ svn commit
Do the same for our centrepoint project and set the appropriate SCM URLs in its POM.
(Don't forget to commit the changes you made.)
Add the effectivemaven-parent project in Continuum again and... success! We now see it being added and after a while, we are directed to the Project Group Summary page.
As we can see from the screenshot above, most of the information in the Project Group were captured from the POM. The Project Group Id of the created project group was from the
Continuum groups its projects making it easier and far more flexible to build a set of projects with just one click. This is designed to handle cases such as multi-modules. We can build the entire project tree by building the group as opposed to triggering the build for each one. When a build is triggered at the group level, all the projects underneath it will be built sequentially with respect to their inter-dependencies.
Now, let's add our centrepoint application. Go back to the Add Maven 2.0+ Project page, and add this POM Url file:///path/to/local/checkout/of/centrepoint/pom.xml. After it is added, we should see the project's Project Group Summary as follows:
As centrepoint has a different groupId from its parent effectivemaven-parent, it was added as a separate Project Group.
Project groups are especially handy when we have a huge project with a lot of components that may be released independently. As releases are done by Project Group in Continuum, we can add each component as a separate group so we can easily release that component without the need to release the entire project.
The build queues
In Continuum, when a project build is triggered—either forced or from the schedule, the project is added to a build queue. This build queue can be viewed via the Queues page. The Queues page is divided into six sections, as we can see below:
The Current Build section contains the project currently being built. The Build Queue, on the other hand, contains the projects waiting in line to be built next. The Build Queue column in each section shows in which Build Queue the project is going to be built.
The Current Checkout section contains the project currently being checked out from the source repository. This would have a value, if and only if a new project was just added to Continuum or a project which was set to be built fresh (checkout a fresh copy of the project) each time was triggered to be built. The Checkout Queue contains the projects waiting in line to be checked out from the source repository.
The Current Prepare Build section contains the project being prepared to be built. When a project is in the prepare build stage, its build state and working directory (if it already exists) is updated, making it ready for building. Once it is out of the prepare build phase, the project will be added either to the Checkout Queue or the Build Queue, depending on whether the project has already been checked out or not. The Prepare Build Queue on the other hand, just lists the projects waiting to be prepared for building.
After a project has finished building, it will be removed from the Queues page. Administrators can access queues to stop the current build or remove builds from the queue.
If you have read this article you may be interested to view :