Icinga is a scalable open source monitoring system that keeps a close watch on our network and server infrastructure, alerts us of problems and resolutions, and gives uptime reports. Icinga is a fork of the popular Nagios project, which aims to build many necessary features on top of it. Icinga is backward compatible with Nagios, so all of the Nagios configuration and plugins can be reused with Icinga as is.
This chapter covers a brief overview of the Icinga architecture and a basic setup to quickly start monitoring a localhost. We will look into the Icinga installation and gain some insight into the initial configuration, so as to get a basic deployment up and running. At the end of this chapter, we will have a web interface showing the status of various services on a localhost and also through an e-mail alert if one of the services goes down.
Icinga consists of various components. The Icinga Core handles scheduling of checks and processing their results. Service states are determined by the Core, by either running checks periodically (active checks) or checking the results that are being reported by a remote system (passive checks). Depending on the checks results, the service state is determined, and depending on the notification configuration, notifications are sent if there is a change in the service state.
The Core is not aware of the checks being executed, nor the notification method that is being used. It simply forks the processes to execute the check plugins, and also performs the processing of exit codes. The notification methods are defined by commands that invoke the underlying operating system provided commands (such as sendmail), or scripts that use notification services to actually deliver SMS, IMs, and so on; this means that Icinga is not aware of the particular notification methods either, it is only aware of the command/script to invoke or sending the notification.
The Core also provides all sorts of information (downtimes, check results, alert history, command execution logs, and so on) via logfiles, which can be used to generate reports on a web interface, for instance. Custom scripts and add-ons can use this logfile to read the current state, and data can show this information in various ways, which is useful for analysis, writing add-ons, and so on.
The data revealed by Icinga can be used to build various web interfaces to present monitoring state, and to do various actions from the web interface (schedule downtime, add comments, disable checks, and so on). There are already a few web interfaces available (Icinga Classic, Thruk, and so on) that can be plugged into Icinga and used.
There are many add-ons available online that extends Icinga's functionality to suit specific needs. Some available add-ons include NagVis (visualizing monitoring status), PNP4Nagios (graphing), and NConf and NagiosQL (managing configuration from web interface) . You can find a number of add-ons by visiting http://www.monitoringexchange.org/.
Icinga's configuration is stored in regular text files along with configuration objects in a simple-to-understand format. Each target server, checks to run on them, and so on are configuration objects that are related to each other as desired.
It is good to keep the outcome of the installation process in mind before beginning to have an overall idea of where we are headed, and what things will look like when we are done. We should expect the following at the end this chapter:
Icinga installed with default configuration to monitor itself
Icinga has started to monitor the localhost with most common checks (load, disk, number of logged-in users, HTTP, ping, SSH, swap, number of processes, and so on)
Icinga web UI showing us the status of above checks
E-mail notifications when a check fails
If you are building Icinga from the source, you should have the GCC compiler and some libraries; Icinga will prompt for these during compilation if they are not found. Apart from that, you would need to have a functioning web server for the web interface.
We will need the following before we can start with the installation:
GCC compiler
Apache web server
Nagios plugins (
nagios-pluginspackage for most distributions)SMTP relay server (for example, Postfix) with proper configuration so that
mail/mailxcommands work
After we have these, we can move on to the download and installation steps.
There are various sources and types of Icinga installers. You can get distribution-specific packages, like DEBs for Debian/Ubuntu, RPMs for Red Hat/CentOS, and so on. You can also get the source code and build packages yourself, or directly compile and install it. The source tarball can be downloaded from Icinga's official website from its Downloads section.
The recommended way of installing Icinga is by distribution packages. Debian (Squeeze/Wheezy) and Ubuntu already have upstream packages available on Launchpad, but the latest stable version may be old (Icinga is still relatively under development, so the latest version is preferred since it will have a lot of bug fixes). Red Hat/CentOS have Icinga packages located in the RepoForge YUM repository (http://repoforge.org/). It may still be useful to build an RPM for yourself, for using backported bug fixes without having to wait for the next release. Icinga's default source tarball already has the SPEC file for RPM creation.
This is optional and should be done only if you want to build your own RPM. Otherwise, use the RepoForge repository mentioned previously and skip this section.
Icinga's source tarball has the SPEC file for building RPMs. The procedure to build RPMs remains as usual. Create a non-root user account, and log in with that user to build the RPMs. Do not do this as the root user.
Download the Icinga source tarball and extract it. Install RPM Development Tools using the following command:
$ sudo yum install rpmdevtools
You can also install it using the following command:
$ sudo yum install @development-tools fedora-packager
Now, set up the build folder:
$ rpmdev-setuptree $ ls ~/rpmbuild BUILD RPMS SOURCES SPECS SRPMS
Now, copy the SPEC file from the extracted source to SPECS and the source tarball to SOURCES:
$ cp icinga.spec ~/rpmbuild/SPECS $ cp icinga-*.tar.gz ~/rpmbuild/SOURCES
Build the final RPM packages using the following commands:
$ cd ~/rpmbuild $ rpmbuild -bb SPECS/icinga.spec
The preceding command might give some dependency errors. Install the listed packages with yum, and run the last rpmbuild command again. If all the dependencies are fixed, the compilation should start, which usually takes some time to complete. The RPM package will be created inside the RPMS/$arch folder, where $arch is the server's architecture (i386/x86_64).
$ ls ~/rpmbuild/RPMS/x86_64 icinga-1.9.1-1.el6.x86_64.rpm icinga-idoutils-1.9.1-1.el6.x86_64.rpm icinga-devel-1.9.1-1.el6.x86_64.rpm icinga-idoutils-libdbi-mysql-1.9.1-1.el6.x86_64.rpm icinga-doc-1.9.1-1.el6.x86_64.rpm icinga-idoutils-libdbi-pgsql-1.9.1-1.el6.x86_64.rpm icinga-gui-1.9.1-1.el6.x86_64.rpm
The icinga package is the Icinga Core; icinga-doc provides the Icinga offline documentation; icinga-gui provides the Icinga web interface; and icinga-idoutils-* provides an optional component to store Icinga information in a database. Now, install the icinga, icinga-doc, and icinga-gui built RPMs:
$ cd ~/rpmbuild/RPMS/x86_64/ $ sudo rpm –ivh {icinga,icinga-doc,icinga-gui}-*.rpm
The installation should succeed. Now install the Apache and Nagios plugins. The nagios-plugins package provides checking of all the basic check plugins that are commonly used for monitoring:
$ sudo yum install httpd nagios-plugins
Also, install SMTP relay server (Postfix is used in the next example) and use the mail command for sending e-mail alerts:
$ sudo yum install postfix mailx
Debian packages for Ubuntu are available at https://launchpad.net/ubuntu/+source/icinga, which can be used to install Icinga on an Ubuntu Server.
$ sudo apt-get install icinga icinga-doc
This command should install the Icinga package which is ready for use. Install the Apache and Nagios plugins:
$ sudo apt-get install apache2 nagios-plugins
Also use the postfix and mail commands:
$ sudo apt-get install postfix bsd-mailx
This setup is good enough for now. Let's continue with the next section.
Let us see how to compile Icinga from the source using the following steps:
Download the Icinga source tarball and extract it:
$ tar zxvf icinga-1.9.1.tar.gz $ cd icinga-1.9.1
Run the configure script:
$ ./configure --prefix=/usr --disable-idoutilsIDOUtils is an optional module that comes with Icinga but we don't want to use it, so we use the
configurescript to disable the module compilation.Compile the source code:
$ make allThis will take some time. After it is done, it's time to install!
$ sudo make fullinstall $ sudo make install-config
Icinga is now installed and (almost) ready for use.
We need to create a user named icinga because the Icinga server will run as this user:
$ useradd –m icinga $ passwd icinga
Since the web server will be run as the apache user (in most cases, this is by default, and is not changed unless required by other applications using Apache), we need to add this user to the icinga group.
$ usermod –a icinga apache
The default web server user may be different for different distributions (for example, www, and so on).
As mentioned in the Requirements section, make sure you have a proper SMTP relay server (such as Postfix) setup so that the usual mail/mailx commands can work. Verify this by sending yourself a test e-mail with the following command on the server (make sure you replace [email protected] with your own e-mail address):
$ echo "This is a test email" | /bin/mail –s "Test email" [email protected]
Proceed further if it works and you receive an e-mail. Troubleshoot your SMTP server if the preceding command gives an error, or you don't receive an e-mail. Open /etc/icinga/objects/contacts.cfg, and replace [email protected] with your own e-mail address in following line:
email [email protected]
By default, notifications are disabled, so we need to enable them first. Open /etc/icinga/objects/localhost.cfg, and comment all occurrences of the following line:
notifications_enabled 0
The configuration files can be commented with a hash (#) or a semicolon (;):
# notification_enabled 0
After the installation step, we now have Icinga Core, the web UI, and the Nagios plugins in place. The configuration to monitor your localhost for common services such as ping check, system load, and disk space is already in place. Now, start the icinga service, and (re)start the apache web server to begin the monitoring and see what's happening in the web interface. This can be done using the following commands.
For RedHat/CentOS/Fedora:
$ service icinga start $ service httpd start
For Ubuntu/Debian:
$ /etc/init.d/icinga start $ /etc/init.d/apache2 start
Make sure that there were no errors reported for the preceding commands. You can now access the web interface at http://localhost/icinga. The default authentication credentials are username: icingaadmin and password: icingaadmin.
Tip
If you get a connection refused error, make sure that Apache was started properly. Also check if you have proper firewall settings in place to allow connections to the web server. This should not be a problem if you are already using the Apache server for other purposes.
The web server may give constant "Internal Server Errors", due to some distributions shipping with SELinux which is enabled by default. We need to disable it to have the web functioning properly.
$ setenforce 0
You can change the default password for the icingaadmin user to something of your choice with following command:
$ htpasswd /etc/icinga/passwd icingaadmin
This command will prompt for a new password to be set, type in the password and press Enter, this will save your password. Simply reload the web page, and it will ask for the new password.
You should have an interface similar to the following screenshot:
Icinga Web
Go to the Service Detail link on the left sidebar to see all the hosts (localhost only for now) and the services being monitored. It may take a few minutes for Icinga to schedule and complete the checking of all the services; after which, all of them will go green (given that they all pass, of course).
The next critical part is alerting. Note that we have a HTTP check for localhost (visible on the web interface). You can try to stop the web server and see if you get an e-mail notification. Note that it may take five to ten minutes before you get an e-mail alert. Icinga makes sure that the service is really down, by checking multiple times, before it sends out alerts. Also note that when we stop the web server, the web interface will remain inaccessible for that duration.
At this point, we're done with setting up a very basic monitoring server. The last section before we summarize the chapter provides some insight into the basic configuration options of Icinga, which typically applies to how Icinga operates rather than to what it monitors.
Configuration is central to customizing Icinga to fit your needs. It becomes possible to highly customize Icinga since the configuration is completely text-based. There is no out of the box configuration that you can install, and automatically start discovering servers on your infrastructure and monitoring them. You need to carefully add all such configurations for Icinga to monitor your network efficiently. This topic gives you a brief overview on the Icinga configuration.
The Icinga configuration is split into two main types: resource files and object files, both of which are specified in the main configuration file along with other Icinga-wide options. This main file contains directives that determine how Icinga operates. The resource file contains Icinga macros that are user defined, such as paths to custom plugins, passwords, and so on. The object files contain Icinga object definitions, each object corresponding to a host, service, and so on. Icinga has object types (host, service, command, and so on) with parameters to build the monitoring configuration.
All of the configuration files are located inside the /etc/icinga folder. The commonly used main configuration file is named as icinga.cfg, the resource file is resources.cfg (it can be used more than once and is specified in the main configuration file). Object files are under the /etc/icinga/objects folder, which contains object definitions that specify monitoring targets. The names of the object files can be anything, with the content being object definitions, which we will look into later. Following is what the configuration structure looks like:
$ tree /etc/icinga /etc/icinga ├── cgiauth.cfg ├── cgi.cfg ├── conf.d ├── icinga.cfg ├── modules ├── objects │ ├── commands.cfg │ ├── contacts.cfg │ ├── localhost.cfg │ ├── notifications.cfg │ ├── printer.cfg │ ├── switch.cfg │ ├── templates.cfg │ ├── timeperiods.cfg │ └── windows.cfg ├── passwd └── resource.cfg
We will look at what each configuration file refers to in the subsequent chapters. You can look up the Icinga documentation at http://docs.icinga.org/latest/en/configmain.html for the main configuration file, to see the list of options that are available to customize.
In this chapter, we looked into the basic overview of Icinga's monitoring architecture, installation, and running Icinga's server and web interface. We also looked into the overview of various aspects of configurations that exist in Icinga. Play around with the web interface and the configuration files to get comfortable with Icinga.
In the next chapter, we will go through and understand how the current configuration to monitor the localhost that is in place works.
