Nagios Core Administration Cookbook

You'll need a working Nagios Core 3.0 or greater installation with a web interface, with all the Nagios Core Plugins installed. If you have not yet installed Nagios Core, then you should start with the QuickStart guide: http://nagios.sourceforge.net/docs/3_0/quickstart.html.

We'll assume that the configuration file that Nagios Core reads on startup is located at /usr/local/nagios/etc/nagios.cfg, as is the case with the default install. It shouldn't matter where you include this new host definition in the configuration, as long as Nagios Core is going to read the file at some point, but it might be a good idea to give each host its own file in a separate objects directory, which we'll do here. You should have access to a shell on the server, and be able to write text files using an editor of your choice; I'll use vi. You will need root privileges on the server via su or sudo.

You should know how to restart Nagios Core on the server, so that the configuration you're going to add gets applied. It shouldn't be necessary to restart the whole server to do this! A common location for the startup/shutdown script on Unix-like hosts is /etc/init.d/nagios, which I'll use here.

You should also get the hostname or IP address of the server you'd like to monitor ready. It's good practice to use the IP address if you can, which will mean your checks keep working even if DNS is unavailable. You shouldn't need the subnet mask or anything like that; Nagios Core will only need whatever information the PING tool would need for its own check_ping command.

Finally, you should test things first; confirm that you're able to reach the host from the Nagios Core server via PING by checking directly from the shell, to make sure your network stack, routes, firewalls, and netmasks are all correct:

tom@olympus:~$ ping 10.128.0.21
PING sparta.naginet (10.128.0.21) 56(84) bytes of data.
64 bytes from sparta.naginet (10.128.0.21): icmp_req=1 ttl=64 time=0.149 ms

We can create the new host definition for sparta.naginet as follows:

If the server restarted successfully, the web interface should show a brand new host in the Hosts list, in PENDING state as it waits to run a check that the host is alive:

In the next few minutes, it should change to green to show that the check passed and the host is UP, assuming that the check succeeded:

If the test failed and Nagios Core was not able to get a PING response from the target machine after three tries, for whatever reason, then it would probably look similar to the following screenshot:

The configuration we included in this section adds a host to Nagios Core's list of hosts. It will periodically check the host by sending a PING request, checking to see if it receives a reply, and updating the host's status as shown in the Nagios Core web interface accordingly. We haven't defined any other services to check for this host yet, nor have we specified what action it should take if the host is down. However, the host itself will be automatically checked at regular intervals by Nagios Core, and we can view its state in the web interface at any time.

The directives we defined in the preceding configuration are explained as follows:

Note that we added the definition in its own file called sparta.naginet.cfg , and then referred to it in the main nagios.cfg configuration file. This is simply a conventional way of laying out hosts, and it happens to be quite a tidy way to manage things to keep definitions in their own files.

There are a lot of other useful parameters for hosts, but the ones we've used include everything that's required.

While this is a perfectly valid way of specifying a host, it's more typical to define a host based on some template, with definitions of how often the host should be checked, who should be contacted when its state changes and on what basis, and similar properties. Nagios Core's QuickStart sample configuration defines a simple template host called generic-host, which could be used by extending the host definition with the use directive:

define host {
    use                 generic-host
    name                sparta
    host_name           sparta.naginet
    address             10.128.0.21
    max_check_attempts  3
    contacts            nagiosadmin
}

This uses all the parameters defined for generic-host, and then adds on the details of the specific host that needs to be checked. Note that if you use generic-host, then you will need to define check_command in your host definition. If you're curious to see what's defined in generic-host, then you can find its definition in /usr/local/nagios/etc/objects/templates.cfg.

You'll need a working Nagios Core 3.0 or greater installation with a web interface, all the Nagios Plugins installed, and at least one host defined. If you need to set up a host definition for your web server first, then you might like to read the Creating a new network host recipe in this chapter, for which the requirements are the same.

It would be a good idea to test that the Nagios Core server is actually able to contact the web server first, to ensure that the check we're about to set up should succeed. The standard telnet tool is a fine way to test that a response comes back from TCP port 80 as we would expect from a web server:

tom@olympus:~$ telnet sparta.naginet 80
Trying 10.128.0.21...
Connected to sparta.naginet.
Escape character is '^]'.

We can create the service definition for sparta.naginet as follows:

If the server restarted successfully, the web interface should show a new service under the Services section, in PENDING state as the service awaits its first check:

Within a few minutes, the service's state should change to OK once the check has run and succeeded with an HTTP/1.1 200 OK response, or similar:

If the check had problems, perhaps because the HTTP daemon isn't running on the target server, then the check may show CRITICAL instead. This probably doesn't mean the configuration is broken; it more likely means the network or web server isn't working:

The configuration we've added adds a simple service check definition for an existing host, to check up to three times whether the HTTP daemon on that host is responding to a simple HTTP/1.1 request. If Nagios Core can't get a response to its check, then it will flag the state of the service as CRITICAL, and will try again up to two more times before sending a notification. The service will be visible in the Nagios Core web interface and we can check its state at any time. Nagios Core will continue testing the server on a regular basis and flagging whether the checks were successful or not.

It's important to note that the service is like a property of a particular host; we define a service to check for a specific host, in this case, the sparta.naginet web server. That's why it's important to get the definition for host_name right.

The directives we defined in the preceding configuration are as follows:

Note that we added the service definition in the same file as defining the host, and directly after it. We can actually place the definition anywhere we like, but this happens to be a good way of keeping things organized.

The service we've set up to monitor on sparta.naginet is an HTTP service, but that's just one of many possible services we could monitor on our network. Nagios Core defines many different commands for its core plugin set, such as check_smtp, check_dns, and so on. These commands, in turn, all point to programs that actually perform a check and return the results to the Nagios Core server to be dealt with. The important thing to take away from this is that a service can monitor pretty much anything, and there are hundreds of plugins available for common network monitoring checks available on the Nagios Exchange website: http://exchange.nagios.org/.

There are a great deal more possible directives for services, and in practice it's more likely for even simple setups that we'll want to extend a service template for our service. This allows us to define values that we might want for a number of services, such as how long they should be in a CRITICAL state before a notification event takes place and someone gets contacted to deal with the problem.

One such template that Nagios Core's default configuration defines is called generic-service, and we can use it as a basis for our new service by referring to it with the use keyword:

define service {
    use                    generic-service
    host_name              sparta.naginet
    service_description    HTTP
    check_command          check_http
}

This may work well for you, as there are a lot of very sensible default values set by the generic-service template, which makes things a lot easier. We can inspect these values by looking at the template's definition in /usr/local/nagios/etc/objects/templates.cfg. This is the same file that includes the generic-host definition that we may have used earlier.

You should have a working Nagios Core 3.0 or greater server running, with a web interface and at least one host to check. If you need to do this first, see the Creating a new network host recipe in this chapter.

For this particular kind of contact, you'll also need to have a working SMTP daemon running on the monitoring server, such as Exim or Postfix. You should verify that you're able to send messages to the target address, and that they're successfully delivered to the mailserver you expect.

We can add a simple new contact to the Nagios Core configuration as follows:

With this done, the next time our host changes its state, we should receive messages similar to the following:

When the host becomes available again, we should receive a recovery message similar to the following:

If possible, it's worth testing this setup with a test host that we can safely bring down and then up again, to check that we receive the appropriate notifications.

This configuration adds a new contact to the Nagios Core configuration, and references it in one of the hosts as the appropriate contact to use when the host has problems.

We've defined the required directives for the contact, and a couple of others as follows:

Note that we placed the definition for the contact in contacts.cfg, which is a reasonably sensible place. However, we can place the contact definition in any file that Nagios Core will read as part of its configuration; we can organize our hosts, services, and contacts any way we like. It helps to choose some sort of system, so we can easily identify where definitions are likely to be when we need to add, change, or remove them.

If we define a lot of contacts with similar options, it may be appropriate to have individual contacts extend contact templates, so that they can inherit those common settings. The QuickStart Nagios Core configuration includes such a template, called generic-contact. We can define our new contact as an extension of this template, as follows:

define contact {
    use    generic-contact
    alias  Administrator of sparta.naginet
    email  nagios@sanctum.geek.nz
}

To see the directives defined for generic-contact, you can inspect its definition in the /usr/local/nagios/etc/objects/templates.cfg file.

You should have a working Nagios Core 3.0 or better server running.

We can verify the Nagios Core configuration as follows:

The configuration is parsed as though Nagios Core were about to start up, to check that the configuration all makes sense. It will run basic checks such as looking for syntax errors, and will also check things like having at least one host and service to monitor. Some of the things it reports are warnings, meaning that they're not necessarily problems; examples include hosts not having any services monitored, or not having any contacts defined.

This is the quickest way to get an idea of whether the Nagios Core configuration is sane and will work correctly. Whenever there's trouble restarting the Nagios Core server, it's a good idea to check the output of this command first. In fact, it's a good habit to check the configuration before restarting, particularly if we're unsure about the configuration changes, or if the monitoring server is checking something very important! This means if it turns out that our configuration is broken, then the Nagios Core daemon will keep running with the configuration from the point before we changed it, and we can fix things before we restart.

The program at /usr/local/nagios/bin/nagios is actually the same program that runs the Nagios Core server, but the -v part of the command is a switch for the program that verifies the configuration instead, and shows any problems with it. The second path is to the configuration file with which Nagios Core starts, which in turn imports the configuration files for objects, such as contact, host, and service definitions.

You should have a working Nagios Core 3.0 or better server running, with a web interface.

You should also have at least two hosts that form a meaningful group; perhaps they're similar kinds of servers, such as webservers, or are monitored by the same team, or all at a physical location.

In this example, we have two webservers, sparta.naginet and athens.naginet, and we're going to add them to a group called webservers.

We can add our new hostgroup webservers to the Nagios Core configuration as follows:

We should now be able to visit the Host Groups section of the web interface, and see a new hostgroup with two members:

The configuration we added includes a new file with a new hostgroup into the Nagios Core configuration, and inserts the appropriate hosts into the group. At the moment, all this is doing is creating a separate section in the web interface for us to get a quick overview of only the hosts in that particular group.

The way we've added hosts to groups is actually not the only way to do it. If we prefer, we can name the hosts for the group inside the group definition, using the members directive, so that we could have a code snippet similar to the following:

define hostgroup {
    hostgroup_name  webservers
    alias           Webservers with Greek names
    members         athens.naginet,sparta.naginet
}

This extends to allowing us to make a hostgroup that always includes every single host, if we find that useful:

define hostgroup {
    hostgroup_name  all
    alias           All hosts
    members         *
}

If we're going to use hostgroups extensively in our Nagios Core configuration, then we should use whichever method is going to be easiest for our configuration. We can use both, if necessary.

It's worth noting that a host can be in more than one group, and there is no limit on the number of groups we can declare, so we can afford to be quite liberal with how we group our hosts into useful categories. Examples could be organizing servers by function, manufacturer, or colocation customer, or routers by BGP or OSPF usage; it all depends on what kind of network we're monitoring.

You should have a working Nagios Core 3.0 or better server running, with web interface.

You should also have at least two services defined that form a meaningful group; perhaps they're similar kinds of services, such as mail services, or are monitored by the same team, or all on the same set of servers at a physical location.

In this example, we have three servers performing mail functions: smtp.naginet, pop3.naginet, and imap.naginet, running an SMTP, POP3, and IMAP daemon, respectively. All three of the hosts are set up in Nagios Core, and so are their services. We're going to add them into a new servicegroup called mailservices.

Here are the definitions of the hosts and services used in this example, so you can see how everything fits together:

define host {
    use                 linux-server
    host_name           smtp.naginet
    alias               smtp
    address             10.128.0.31
    hostgroups          webservers
}
    define service {
        use                  generic-service
        host_name            smtp.naginet
        service_description  SMTP
        check_command        check_smtp
    }
define host {
    use                 linux-server
    host_name           pop3.naginet
    alias               pop3
    address             10.128.0.32
    hostgroups          webservers
}
    define service {
        use                  generic-service
        host_name            pop3.naginet
        service_description  POP3
        check_command        check_pop
    }
define host {
    use                 linux-server
    host_name           imap.naginet
    alias               imap
    address             10.128.0.33
    hostgroups          webservers
}
    define service {
        use                  generic-service
        host_name            imap.naginet
        service_description  IMAP
        check_command        check_imap
    }

We can add our new servicegroup with the following steps:

We should now be able to visit the Service Groups section of the web interface, and see a new servicegroup with three members:

The configuration we added includes a new file with a new servicegroup into the Nagios Core configuration, and inserts the appropriate services into the group. This creates a separate section in the web interface for us to get a quick overview of only the services in that particular group.

The way we've added services to the groups is actually not the only way to do it. If we prefer, we can name the services (and their applicable hosts) for the group inside the group definition, using the members directive, so that we could have a code snippet similar to the following:

define servicegroup {
    servicegroup_name  mailservices
    alias              Mail services
    members            smtp.naginet,SMTP,pop3.naginet,POP3
}

Note that we need to specify both the host that the service is on, and then the services to monitor on it, comma-separated. The hostname comes first, and then the service.

This extends to allowing us to make a servicegroup that always includes every single service, if we find that useful:

define servicegroup {
    servicegroup_name  all
    alias              All services
    members            *
}

If we're going to be using servicegroup definitions extensively in our Nagios Core configuration, we should use whichever of the two methods to add services to groups that we think is going to be easiest for us to maintain.

It's worth noting that a service can be in more than one group, and there is no limit on the number of groups we can declare, so we can afford to be quite liberal with how we group our services into categories. Examples could be organising services by the appropriate contact for their notifications, for internal functions, or customer facing functions.

You should have a working Nagios Core 3.0 or better server running.

You should also have at least two contacts that form a meaningful group. In this case, we have two staff members, John Smith and Jane Doe, who are both a part of our network operations team. We want both of them to be notified for all the appropriate hosts and services, so we'll add them to a group called ops. Here are the definitions with which we're working:

define contact {
    use           generic-contact
    contact_name  john
    alias         John Smith
    email         john@naginet
}
define contact {
    use           generic-contact
    contact_name  jane
    alias         Jane Doe
    email         jane@naginet
}

We can create our new ops contactgroup as follows:

With this group set up, we are now able to use it in the contactgroups directive for hosts and services, to define the contacts to which notifications should be sent. Notifications are sent to all the addresses in the group. This can replace the contacts directive where we individually name the contacts.

This means, for example, that instead of having a service definition similar to the following:

define service {
    use                  generic-service
    host_name            sparta.naginet
    service_description  HTTP
    check_command        check_http
    contacts             john,jane
}

We could use the following code snippet:

define service {
    use                  generic-service
    host_name            sparta.naginet
    service_description  HTTP
    check_command        check_http
    contact_groups       ops
}

If John Smith were to leave the operations team, then we could simply remove his contact definition, and nothing else would require changing; from then on, only Jane Doe would receive the service notifications. This method provides a layer of abstraction between contacts and the hosts and services for which they receive notifications.

You should have a working Nagios Core 3.0 or better server running.

We can set up our new time period, which we'll call weekdays, as follows:

In our host and service definitions, there are two directives, check_period and notification_period. These directives are used to define the times during which a host or service should be checked, and the times when notifications about them should be sent. The 24x7 and workhours periods are defined in the timeperiods.cfg file that we just edited, and are used in several of the examples and templates.

We've just added two more of these time periods, which we can now use in our definitions for hosts and services. The first is called weekdays, and corresponds to any time during a weekday; the second is called weekends, and corresponds to any time that's not a weekday. Note that in both cases, we specified the dates and times by naming each individual day, and the times to which they corresponded.

The definitions for dates are quite clever, and can be defined in a variety of ways. The following are all valid definitions for days and time periods:

It's likely that the standard 24x7 and workhours definitions will be fine for day-to-day monitoring, maybe with a weekdays and weekends definition added. However, there may well come a time when we need a specific host or service monitored on an unusual schedule, particularly if we're debugging a specific problem that only manifests around a certain time, or have a lot of contacts to manage, or a complex on-call roster.

Note that Nagios Core can behave in unusual ways, particularly with uptime reporting, if the time periods for our monitoring of hosts and services don't add up to 24 hours. Ideally, we should check and notify all our hosts and services in at least some way around the clock, but dealing with the notifications in different ways depending on schedule; for example, we could page the systems administrators about a non-critical system during work hours, but just e-mail them when they're asleep!

You should have a working Nagios Core 3.0 or better server running, with a web interface. You should be familiar with adding services to individual hosts.

You should also have at least one hostgroup defined, with at least one host in it; we'll use a group called webservers, with the hosts sparta.naginet and athens.naginet defined in it.

For reference, here is the hostgroup definition and the definitions for the two hosts in it:

define hostgroup {
    hostgroup_name  webservers
    alias           Webservers
}
define host {
    use         linux-server
    host_name   athens.naginet
    alias       athens
    address     10.128.0.22
    hostgroups  webservers
}
define host {
    use         linux-server
    host_name   sparta.naginet
    alias       sparta
    address     10.128.0.21
    hostgroups  webservers
}

We can create the service definition for the webservers group as follows:

It's important to note that if we are already monitoring those hosts with a per-host service of the same name, then we will need to remove those definitions as well; Nagios Core may not start if a service of the same description is already defined on the same host.

Adding a service to a hostgroup works in exactly the same way as adding it to an individual host, except that it only requires one definition, which is then individually applied to all the hosts in the group. This means it's a very good way to keep a Nagios Core configuration tidier. If we have a group of 50 different web servers in it and we need to monitor their HTTP services on the same basis for each one of them, then we don't need to create 50 service definitions; we can just create one for their hostgroup, which amounts to a smaller and more easily updated configuration.

Like the host_name directive for services, the hostgroup_name directive can actually have several hostgroups defined, separated by commas. This means that we can apply the same service to not just one group, but several. For services that we would want to run on several different groups (for example, basic PING monitoring) this can amount to a much more flexible configuration.