Instant Nagios Starter

Before you install Nagios, you will need to check that you have all of the required elements, listed as follows:

The following code example will describe how to install prerequisite packages for both CentOS/RHEL-based distributions, and create users that are necessary for Nagios to run.

Run the following command from a command-line shell to install required packages for source installation:

yum install -y autoconf wget httpd php gcc glibc glibc-common gd gd-devel make net-snmp xinetd openssl-devel unzip

Now let's set up user permissions for the Nagios process using the following commands:

useradd nagios
groupadd nagcmd
usermod -a -G nagcmd nagios

The easiest way to download Nagios is as a compressed tarball from http://www.nagios.org/download.

The following example demonstrates downloading the latest packages at the time of writing this book. We suggest that you download the most current stable build of both the Nagios Core and Nagios plugins tarballs. Lets get both of these packages downloaded and extracted into the /tmp directory:

cd /tmp
wget http://prdownloads.sourceforge.net/sourceforge/nagios/nagios-3.5.0.tar.gz
wget http://prdownloads.sourceforge.net/sourceforge/nagiosplug/nagios-plugins-1.4.16.tar.gz
tar zxf nagios-3.5.0.tar.gz
tar zxf nagios-plugins-1.4.16.tar.gz

A typical Nagios installation is made up of more than just the Nagios Core monitoring engine. The list of available add-ons for Nagios is quite large, but the following steps will guide you through the installation and setup of the key packages required for almost all Nagios installations. The following steps will guide you through installing Nagios Core, Nagios plugins, NRPE for remote agent monitoring, and NRDP for passive monitoring.

cd /tmp/nagios
./configure -–with-command-group=nagcmd
make all
make install
make install-init
make install-config
make install-commandmode
make install-webconf

cd /tmp/nagios-plugins-1.4.16
./configure
make
make install

/usr/local/nagios/bin/nagios -v /usr/local/nagios/etc/nagios.cfg
/etc/init.d/nagios start

The locations for Nagios files will vary depending on whether or not you've installed from source or a packaged version, and most Nagios add-ons require knowledge of where many of these key files are located. The following list outlines some key files and locations in your Nagios installation for a source installation:

The preceding steps are required for all the Nagios installations to function at any level. However, most environments will require monitoring data for information that cannot be accessed externally on a host machine. For example, CPU load or memory usage are sets of data that can't be accessed on a remote machine without some sort of agent or internal script that has access to that data. Some users utilize SNMP or WMI to access this data for Nagios, but both of these methods require more advanced setup methods, and don't scale as well as the two methods described in the following sections. The latest versions for both of these add-ons can be obtained from http://exchange.nagios.org.

NRPE

NRPE allows you to remotely execute Nagios plugins as active checks on remote machines by means of an agent. NRPE can communicate with agents on both Windows and Linux/Unix machines. NRPE contacts the agent on the remote machine, asks it to run a check command, and returns the results back to Nagios, as illustrated in the following diagram:

1. The following commands will install the NRPE listener on the system and add it as a sub-service of the xinetd daemon:

   cd /tmp
   wget http://prdownloads.sourceforge.net/sourceforge/nagios/nrpe-2.13.tar.gz
   tar zxf nrpe-2.13.tar.gz
   cd nrpe-2.13
   ./configure –-enable-command-args
   make all
   make install-plugin
   make install-daemon
   make install-xinetd
   make install-daemon-config
   /etc/init.d/xinetd restart
   

2. The following commands make sure that the xinetd service will be running each time we reboot:

   chkconfig –-add xinetd
   chkconfig –-level 35 xinetd on
   

3. The two key configuration files related to NRPE are /etc/xinetd.d/nrpe and /usr/local/nagios/etc/nrpe.cfg. Be sure to review these settings to make sure they are suitable for your environment.

Getting started with NRPE and active checks will be covered in the Top features you need to know about section.

NRDP

NRDP functions as a form of web API to submit both check results and commands using POST and GET over standard HTTP and HTTPS ports. Using NRDP, Nagios acts as a listener for results for a host or service, while cron or some other processes run the check on their own schedule. This is known as a passive check. NRDP allows for batch submissions of passive checks, and uses an XML format for data transport. NRDP is considered to be the most lightweight way to submit passive checks to Nagios.

The following steps will guide you through the installation process for NRDP:

1. To install NRDP, download the latest version from Nagios and extract the files using the following commands:

   cd /tmp
   wget http://assets.nagios.com/downloads/nrdp/nrdp.zip
   unzip nrdp.zip
    Copy files to the installation directory with new permissions.
   mkdir /usr/local/nrdp
   cd nrdp
   cp -r * /usr/local/nrdp
   chown -R nagios.nagios /usr/local/nrdp
   

2. Set up web access to NRDP by adding the apache alias.

   Note

   Note that on Ubuntu/Debian machines, the apache configuration directory is /etc/apache2/conf.d, while RHEL based systems store configuration files in /etc/httpd/conf.d.

   cp nrdp.conf /etc/httpd/conf.d
   /etc/init.d/httpd restart
   

3. Add at least one authorization token for remote senders to submit results or commands. Open the /usr/local/nrdp/server/config.inc.php file in a text editor and add a new token to the authorized_tokens array.

4. The following example shows two tokens added to the authorization array:

   // NOTE: tokens are just alphanumeric strings - make them hard to guess!
   $cfg['authorized_tokens'] = array(
   "asd7fjk3l34",
   "df23m7jadI34",
   );
   

5. Save the config.inc.php file and close it. From here you can test the NRDP API, by accessing it from the following address:

   http://<your_nagios_server_address>/nrdp

   

There are a few remaining steps that should be completed after Nagios is installed to make sure everything is accessible after installation.

/etc/init.d/httpd start
htpasswd -c /usr/local/nagios/etc/htpasswd.users nagiosadmin

chkconfig –-add nagios
chkconfig –-level 35 nagios on
chkconfig –-add httpd
chkconfig –-level 35 httpd on

SELINUX=disabled
SETLOCALDEFS=0

setenforce 0

# HTTP/HTTPS
-A INPUT -m state --state NEW -m tcp -p tcp --dport 80 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 443 -j ACCEPT
# NRPE
-A INPUT -m state --state NEW -m tcp -p tcp --dport 5666 -j ACCEPT
# NSCA
-A INPUT -m state --state NEW -m tcp -p tcp --dport 5667 -j ACCEPT

/etc/init.d/iptables restart

By this point, you should have a working installation of Nagios and you are now free to explore and discover more about it. You can access the Nagios Core interface by entering the following URL in your web browser. Replace the IP address with the IP address or host name of the machine that Nagios is installed on.

http://<your_nagios_server_address>/nagios

Enter nagiosadmin as the username, and the initial password created upon installation to log into the interface.

Nagios knows which configuration files to parse by maintaining a master reference in the nagios.cfg file. For any new configuration file to be recognized by Nagios, either the file or the directory it is in has to be defined in nagios.cfg. For source installations, this file is located at /usr/local/nagios/etc/nagios.cfg. Perform the following steps to modify nagios.cfg:

A host in Nagios is any machine or device with an IP address or a host name. The following example will demonstrate the creation of a basic host configuration file that we can add to the monitoring configuration:

Services in Nagios are processes, applications, metrics, and anything else that can be monitored under the scope of the associated host. The following example creates a basic service definition for the test used, and will be used to start monitoring with a simple HTTP check:

Alerting is an essential part of monitoring infrastructure with Nagios, but it is recommended to minimize or even disable the use of alerts while setting up your monitoring environment. Users who receive too many false alerts will be trained to ignore them. Setting up effective alerting starts with creating appropriate contacts and contact groups for the hosts and services that are being monitored. Contacts also form the basis for host and service permissions in Nagios. A regular-level user in Nagios will only be able to view and submit commands for hosts and services that they are contacts for, unless he/she is granted some level of global access in the cgi.cfg file. The following are the steps to create and assign contacts:

All monitoring and event handling is done based on rules defined in the object configuration files, so the monitoring process requires a valid configuration in order to run. Always verify any configuration changes before attempting to restart Nagios, using the following steps. Attempting to restart Nagios with configuration errors will halt all the monitoring on the system.

Any check that Nagios schedules and executes itself is known as an active check. In the previous section, we covered how to perform a basic check for the HTTP service running on a particular machine. But what happens when we need to monitor something internal, such as CPU load or a process status on another machine? This information is not available for an outside machine to simply request, unless there is some sort of agent established on the remote machine that can retrieve that information.

cd /usr/local/nagios/libexec
./check_nrpe -H 192.168.5.59
NRPE v2.12

command[check_users]=/usr/local/nagios/libexec/check_users -w 5 -c 10
command[check_load]=/usr/local/nagios/libexec/check_load -w 15,10,5 -c 30,25,20

command[<name_of_command>]=/path/to/plugin <arguments>

./check_nrpe -H 192.168.5.59 -c check_users
USERS OK - 1 users currently logged in |users=1;5;10;0

define command {
       command_name    check_nrpe_generic
       command_line    $USER1$/check_nrpe -H $HOSTADDRESS$ -c $ARG1$
}

define host{
       host_name      My_NRPE_Host
       alias          NRPE Test Host
       address        192.168.5.59
       use            linux-server
}
define service{
        use            generic-service       
        host_name            My_NRPE_Host
        service_description   Users
        check_command         check_nrpe_generic!check_users
        }
define service{
        use            generic-service       
        host_name            My_NRPE_Host
        service_description   CPU Load
        check_command         check_nrpe_generic!check_load
        }

command[check_users]=/usr/local/nagios/libexec/check_users $ARG1$

define command {
    command_name   check_nrpe_generic
  command_line   $USER1$/check_nrpe -H $HOSTADDRESS$ -c $ARG1$ -a  '$ARGS2$'
}
define service{
  use           generic-service       
  host_name            My_NRPE_Host
  service_description   Users
  check_command         check_nrpe_generic!check_users!'-w 5 -c 10'
}

 check_nt -H host -v variable -p port [-w warning] [-c critical][-l params] [-d SHOWALL] [-t timeout] [-s password]

./check_nt -H 192.168.5.11 -v CPULOAD -s password -l 5,10,15 -p 12489
CPU Load 2% (5 min average) |   '5 min avg Load'=2%;10;15;0;100
define command{
    command_name    	check_nt
    command_line     	$USER1$/check_nt -H $HOSTADDRESS$ -p 12489 -v $ARG1$ $ARG2$
}


define service{
    use                     generic-service
    host_name               winserver
    service_description     CPU Load
    check_command           check_nt!CPULOAD!-l 5,10,15
}

There are some circumstances where active checks may not be suitable for a host or service that needs to be monitored. Nagios also supports the ability to push results from the client up to the Nagios server, and Nagios merely acts as the listener for check results. This is known as a passive check, since Nagios does not schedule or initiate a check for this host or service, it only processes incoming results. Passive checks are useful for the following situations:

NRDP (Nagios Remote Data Protocol) acts like an HTTP API for submitting check results back to Nagios. It can be used to submit Nagios commands and a batch of check results all at once. NRDP data can be sent with one of the several send_nrdp scripts that exist on exchange.nagios.org, and can be sent from almost any OS platform. At the time of writing this book, send_nrdp scripts exist in Perl, Python, and Shell. The following example illustrates configuration definitions needed for Nagios to be able to process passive checks for a host and service:

define host{
    host_name      My_NRDP_Host
    alias          NRDP Passive Host
    address        192.168.5.59
  use            linux-server
  check_command             check_dummy!1!'Results are stale'
    active_checks_enabled     0
    passive_checks_enabled     1
    check_freshness        1
  freshness_checking_enabled	1
  freshness_threshold      600
}
define service{
    use                generic-service       
    host_name              My_NRDP_Host
    service_description       Passive_Service
  check_command             check_dummy!1!'Results are stale'
    active_checks_enabled     0
    passive_checks_enabled     1
    check_freshness        1
  freshness_checking_enabled  1
  freshness_threshold      10
  }

The previous configuration will implement the following logic for the passive checks:

See the Nagios Core manual on object definitions for a detailed explanation of each configuration directive used for hosts and services from the following link:

http://nagios.sourceforge.net/docs/nagioscore/3/en/objectdefinitions.html

Once a host and service is configured for passive checks, results can be sent via send_nrdp by using the following command:

./send_nrdp.sh -u http://myserver/nrdp -t secrettoken -H My_NRDP_Host -s Passive_Service -S 0 -o 'Everything is OK!'

send_nrdp also supports sending results directly from an XML file by using the -f flag, or a directory of files by passing the -D flag, with the directory path of the temp files. The following example illustrates an example XML file that can be specified for send_nrdp:

<?xml version='1.0'?>
<checkresults>
<checkresult type="host" checktype="1">
<hostname>YOUR_HOSTNAME</hostname>
<state>0</state>
<output>OK|perfdata=1.00;5;10;0</output>
</checkresult>
<checkresult type="service" checktype="1">
<hostname>YOUR_HOSTNAME</hostname>
<servicename>YOUR_SERVICENAME</servicename>
<state>0</state>
<output>OK|perfdata=1.00;5;10;0</output>
</checkresult>
</checkresults>

NRDP is an extremely useful tool for scaling large installations, using flexible check schedules, or monitoring remote locations.

Setting up hosts and services is only half of effective monitoring, and is incomplete without setting up some sort of method for incident management or alerting. I once did a remote session with a user where all of the alerts from Nagios got dumped into a single inbox that had over 30,000 e-mails. During the troubleshooting of a monitoring service migration, I had asked the user if a selection of hosts had always been down on the old server, and the user didn't know, because everyone in their environment had been trained to ignore all of the alerts from Nagios and to simply filter them into an effective trash can. Without alerting, there is no monitoring taking place, so it pays to take the time to set it up properly. Nagios has an enormous amount of flexibility in how alerts and event handling can be set up, and once set up properly, can drastically increase uptime, and improve response times to any problems within an environment. Let's examine a few common scenarios and questions that users want for incident response:

Nagios is an enormous source of information for data analysis and reporting. The scope of Nagios Core is primarily intended for monitoring, alerting and event handling, and although the web interface has several built-in reports such as availability, trends, alert statistics, and the network status map, many users prefer to extend Nagios even further through the use of various add-ons in order to track historical trends in metrics, or to visualize data. Let's first examine some of the built-in reporting tools for Nagios, and then look at a few of the most popular add-ons that are used for reporting and data visualization.

Performance graphing

Most of the check plugins that you'll find with Nagios return performance data, which is some form of performance metric that can be passed to an external graphing tool. Most often this data is passed to an RRD (Round Robin Database) file, which is a binary storage file that never grows in size. Using an add-on project such as PNP4Nagios or Nagiosgraph allows you to store performance metrics in RRD files and view them on a graph for as far back as four years.

More information and documentation for these projects can be found on the following project home pages:

• PNP4Nagios: http://www.pnp4nagios.org/

• Nagiosgraph: http://sourceforge.net/projects/nagiosgraph/

NagVis

NagVis is a visualization add-on for Nagios that allows for custom maps and diagrams to be built, which contain live status information from Nagios. This tool is enormously flexible to create custom visualizations.

NagVis project home can be found at http://www.nagvis.org/.

Nagios BPI

Nagios Business Process Intelligence (Nagios BPI) is a way to visualize and monitor business processes that are made up of many hosts, services, and even other BPI groups. Nagios BPI is used by many users to monitor the real state of the network and allows for rules and thresholds to be defined for business processes, so that their states can be effectively monitored.

The Nagios BPI add-on is available at exchange.nagios.org.

Ndoutils

Ndoutils provides a database backend for all monitoring data that is processed by the Nagios Core monitoring engine. Many users have a need to use their own custom reporting tools, and prefer a database backend for historical information as opposed to mining log archives to create custom reports. Ndoutils supports multiple Nagios servers storing data on a local or remote database server.

The Ndoutils add-on is available on exchange.nagios.org.

Nagios creates unique challenges for troubleshooting and debugging, not only because so much of what it does is tied to the environment that it's installed on, but also because it depends largely on what's being monitored. Knowing where to look for clues and when to ask for help are essential steps to getting started with Nagios. The following list identifies some key places to look for clues in troubleshooting problems with Nagios: