Chapter 11. Automating and Extending Nagios Core

In this chapter, we will cover the following recipes:

• Allowing and submitting passive checks

• Submitting passive checks from a remote host with NSCA

• Submitting passive checks in response to SNMP traps

• Setting up an event handler script

• Tracking host and service states with Nagiosgraph

• Reading status in a MySQL database with NDOUtils

• Reading status from a Unix socket with MK Livestatus

• Writing customized Nagios Core reports

• Getting extra visualizations with NagVis

• Writing custom Nagios Core management scripts

Introduction

In addition to being useful as a standalone monitoring framework, Nagios Core has a modular design that allows both interaction with and extension to other programs and tools. It does so predominantly using its external command file to control the behavior of the server.

One of the most useful ways of interacting with the Nagios Core server in this way is through the use of passive checks or by checking results for hosts and services that are submitted to the server directly, rather than as the result of the server's own active checks.

The simplest application of the idea of passive checks is to monitor a process that might take an indeterminate amount of time to run and hence resists active checking; instead of the service making active checks of its own, the service accepts a check result submitted by another application, perhaps something like a backup script after it has completed its run. These check results can be sent and accepted via an add-on called NSCA (the Nagios Service...

Allowing and submitting passive checks

In this recipe, you'll learn how to configure Nagios Core to accept passive checks for a service. This allows both users and external applications to directly submit the results of checks to Nagios Core, rather than have the application seek them out itself through polling with active checks, which are performed via plugins such as check_http or check_ping.

We'll show you one simple example of a passive check: flagging a service called BACKUP for an existing host. We'll show you how to do this via a web interface, which is very easy, and via the external commands file, which is slightly more complex but much more flexible and open to automation.

The idea is that when a user or process receives confirmation that the backup process on a host has completed correctly, the user or process is able to supply a check result of OK directly to the service without Nagios Core needing to poll anything by itself.

Getting ready

You should be running a Nagios Core 4.0...

Submitting passive checks from a remote host with NSCA

In this recipe, we'll show you how to automate the submission of passive checks by a remote host using the example of a monitored host, ithaca.example.net, to submit a passive check to a Nagios Core server with information about how its BACKUP service is performing.

For example, if the backup process gets completed successfully, we configure the monitored host to submit a passive check result, specifying that the BACKUP service should have the OK status. However, if there were a problem with the backup, the monitored host could send a passive check result with a WARNING or CRITICAL status.

In both cases, Nagios Core does no checking of its own; it trusts the results submitted by its target host.

To accomplish this setup, we'll use the NSCA add-on. We'll install the NSCA server on the Nagios Core server and the NSCA client program send_nsca on the monitored host.

Getting ready

Here, we assume that you have already followed the Allowing and...

Submitting passive checks in response to SNMP traps

In this recipe, you'll learn how to configure Nagios Core to process SNMP traps, which is information sent by monitored network devices to a central monitoring server.

Because SNMP traps often contain useful or urgent information about how a host is working, processing them in at least some way can be very helpful. It can be very helpful particularly for firmware network devices that can't use send_nsca to submit a passive check result in a standard form, as explained in the Submitting passive checks from a remote host with NSCA recipe.

As an example, most SNMP-capable hosts can be configured to send SNMP traps when one of their network interfaces changes state, perhaps due to a pulled network cable. These traps are known as linkUp and linkDown traps. Monitoring this particular kind of trap is especially useful for devices with a large number of interfaces, such as switches or routers.

Keeping track of these events in Nagios Core is valuable...

Setting up an event handler script

In this recipe, you'll learn how to set up an event handler script for Nagios Core. Event handlers are commands that are run on every state change for a host or service (whether for all hosts or services or just particular ones). They are defined in a similar way to notification commands and check commands for plugins.

In this example, we'll implement a simple event handler that writes the date, the host state, and the number of check attempts to a separate file for a single host. This is a simple example to demonstrate the concept; a more practical and complex application for the use of event handlers is given in the Setting up a redundant monitoring host recipe in Chapter 10, Security and Performance.

Getting ready

You will need a server running a Nagios Core 4.0 or higher version. You should be familiar with defining new commands, as per the Creating a new command recipe in Chapter 2, Working with Commands and Plugins, and the Defining a custom notification...

Tracking host and service states with Nagiosgraph

In this recipe, you'll learn how to install and configure Nagiosgraph, which is a program that is integrated with Nagios Core's performance data tools to produce graphs that show long-term information about how checks for hosts and services are performing.

Getting ready

You will need to run a Nagios Core 4.0 or later server. Nagiosgraph will probably still work with older versions of Nagios Core, but the configuration may be slightly different. The INSTALL document included in the source for Nagiosgraph explains the differences in detail.

You should have a thorough understanding of defining hosts, services, and commands, and you should be able to install new software as the root user on the monitoring server. You should also be, at least, familiar with the layout of your Apache HTTPD server on the monitoring system; this recipe will assume that it is installed in /usr/local/apache.

Because Nagiosgraph has many Perl dependencies, you will need...

Reading status in a MySQL database with NDOUtils

In this recipe, you'll learn how to install the NDOUtils extension to Nagios Core in order to have all of Nagios Core's configuration and data written into a MySQL database. This allows us to easily develop custom reports and interfaces for Nagios Core data with languages such as Perl and PHP and their standard interfaces for the popular MySQL server, as opposed to interacting with Nagios Core's own logs or data format. Some plugins such as NagVis use this format to read information about Nagios Core configuration and objects.

Getting ready

You will need a Nagios Core server version 4.0 or later. NDOUtils will probably still get installed and work on older versions of Nagios Core, but the installation process is slightly different; refer to the INSTALL file included in the NDO source for more information on this.

Nagios Core uses its event broker functionality to write information to the socket for the MySQL database to pick up. You will need...

Reading status from a Unix socket with MK Livestatus

The NDOUtils event broker and database system is a useful way of getting data out of the Nagios Core program and its history in order to generate our own reports and displays but it has a few flaws; the MySQL database setup can be complicated and a hassle to maintain, and the database can grow quite large, which makes searching slow and uses a lot of system resources not just for data searching, but for the housekeeping it does in removing old entries.

The MK Livestatus tool, part of Mathias Kettner's Check_MK suite of Nagios Core monitoring tools, provides a way of retrieving data directly from the running Nagios process by writing queries to a Unix socket and by reading results back from it, without requiring a database to do so. Provided our queries do not need to be too sophisticated, this extension can make things a lot faster and less burdensome on system resources, which is why a growing number of administrators and Nagios plugin...

Writing customized Nagios Core reports

In this recipe, we'll explore some simple applications of the NDOUtils database by trying out some queries and changing one of them into both a simple daily e-mail in Perl and also into a PHP-based RSS feed.

Getting ready

This recipe assumes that you have NDOUtils already installed and that your Nagios Core 4.0 or higher server is monitoring at least a few hosts and services so that the queries we try actually return some data. You should also have some means of executing MySQL queries on the database server. The mysql(1) command-line client will work just fine; a web-based tool such as phpMyAdmin might make the data a little easier to explore.

How to do it...

We can explore some queries against the NDOUtils databases as follows:

1. Retrieve the content and date/time of the latest 10 notifications:

   mysql> SELECT start_time, long_output FROM nagios_notifications ORDER BY start_time DESC LIMIT 10;
   

2. Retrieve the content and date/time of the latest 10 host or...

Getting extra visualizations with NagVis

In this recipe, we'll explore how to go beyond the default network map, discussed in Chapter 8, Managing Network Layout, to get a lot of visualization power with the NagVis extension. NagVis can use the MK Livestatus backend to build custom maps in various styles.

NagVis is most likely of interest to you if you're interested in visualizing Nagios data more extensively, particularly if you're having problems with the scalability of the included Nagios Core status map. The default status map works well for smaller networks but can struggle with rendering larger ones in a timely fashion.

A complete survey of NagVis' functions would not be possible in one recipe, but this recipe will walk you through downloading, installing, and configuring the extension to give you a simple automap in order to get you started.

Getting ready

You should have a running Nagios Core server with version 4.0 or later and have the MK Livestatus backend successfully installed. This...

Writing custom Nagios Core management scripts

As established in Chapter 7, Using the Web Interface, some control over the Nagios Core process can be achieved by different areas of a web interface by enabling or disabling notifications or checks, scheduling downtime for hosts or services, or acknowledging existing programs. However, for large networks in particular, we may often need a more flexible approach to run operations on a large number of hosts or services in batches or to disable or enable features from another script, which is cumbersome to manage via the CGI.

For example, if we had a list of more than a hundred hosts in a segment of our network that were going to become inaccessible due to scheduled maintenance at a known time, it would take a very long time for us to set up all the scheduled downtime in the web interface to avoid sending out notifications for those hosts. It would be better to do this via a command line.

In this recipe, we'll install a custom script in /usr/local...