Using Proxies to Monitor Remote Locations with Zabbix 1.8

Exclusive offer: get 50% off this eBook here
Zabbix 1.8 Network Monitoring

Zabbix 1.8 Network Monitoring — Save 50%

Monitor your network hardware, servers, and web performance effectively and efficiently

$26.99    $13.50
by Rihards Olups | March 2010 | Networking & Telephony Open Source

In this article by Rihards Olups, author of Zabbix 1.8 Network Monitoring, we'll explore usage of proxies that collect the data on behalf of the Zabbix server and then transmit it back to the server, which helps when remote locations can't be accessed directly because of firewall concerns. It also reduces the load on the Zabbix server.

When proxies are useful

A server is not always able to connect directly to the monitored machines. We could use active items everywhere (where Zabbix agents connect to a Zabbix server, discussed in Chapter 3), but that would require access from all of the monitored hosts to the Zabbix server. Additionally, that would not work for SNMP, IPMI, and other monitoring methods. This is common when devices from other organizations have to be monitored, or when such restrictions are in place in a large corporate network.

This is where Zabbix proxy comes in. When set up, only connections to the Zabbix server come from the proxy, which, in turn, does all the monitoring on behalf of the Zabbix server.

zabbix

In this example, there's no need for individual connections from Zabbix to individual devices in the remote location or the other way around—single firewall rule to allow Zabbix proxy connections to the Zabbix server is sufficient.

So how does the proxy work? Let's repeat: Only the Zabbix proxy connects to the Zabbix serverv. This bit of information is crucial to understanding how it actually works. When the proxy connects to the server, it requests configuration information about what it has to monitor, then goes and does just that. As data is gathered, proxy sends it back to the server. Does that sound familiar? It should, because that part sounds just like an active agent, except that the proxy can gather data from different systems pretty much like the Zabbix server can. It sounds almost like a super agent. Actually, it is said that the proxy was at first internally named "super agent", which confirms the similarity.

The Zabbix proxy is a very useful feature, the first stable Zabbix version to introduce it was 1.6, back in 2008. Given the usefulness of the Zabbix proxy, it is surprising that for a long time no other solutions provided something comparable. Today, Zabbix proxies continue to lead the way by having and improving features that are considered most useful for remote data collection. As we will see later, there are other benefits that proxies can provide, so let's get to actually setting one up.

Setting up the proxy

W hen setting up the proxy for this exercise, it is suggested you use a separate (physical or virtual) machine. If not possible, you can choose to run proxy on the same machine, as hints will be provided for such a setup as well.

Before proceeding, we will have to compile the proxy itself. When compiling the proxy there are a few mandatory parameters, --enable-proxy being one. Zabbix proxy also needs a database, so you have to include support for one. While it is not suggested to use SQLite on Zabbix itself because of the performance issue, the proxy might be a good candidate. Unless it is going to monitor lots of systems, SQLite's performance can be satisfactory, and it is trivial to set up because Zabbix server creates and populates the database automatically.

Additionally, Zabbix proxy must have support compiled in for things it will be monitoring, like SNMP, IPMI, and web monitoring. Let's say we want to compile Zabbix proxy with SQLite support and the ability to monitor web, SNMP, and IPMI. Obviously, we will need all the relevant support libraries mentioned when we first compiled Zabbix, but this time we'll also need another one—SQLite development headers. On most distributions they will be included in a package named sqlite-devel or similar, so make sure to install this package before proceeding. Then, on the machine where you plan to install Zabbix proxy, enter the directory where the Zabbix installation source has been extracted to and execute:

$ ./configure --enable-proxy --with-sqlite3 --with-libcurl --with-netsnmp
--with-openipmi && make

This will both configure and compile Zabbix proxy. We can again use the simple and universal solution to create a more or less proper package by executing as root:

# checkinstall --nodoc --install=yes -y --pkgname=zabbix-proxy

The proxy binary is now compiled and in place, we can put our hands on the configuration now. First things first—it must be placed in the correct location:

# cp misc/conf/zabbix_proxy.conf /etc/zabbix

Now open /etc/zabbix/zabbix_proxy.conf as root and make some changes to get the proxy running:

Hostname=proxy

Hostname is an important parameter. As with active agents, if it's wrong, nothing will work.

Server=<Zabbix server IP address>

Zabbix proxy connects to the Zabbix server, so it must know where to connect to.

DBName=/tmp/zabbix_proxy.db

We will try to use an SQLite database, thus the full path to the database file must be specified.

Let's try to start the Zabbix proxy now, as root execute:

# zabbix_proxy

Wait, we did not create the database like we did when installing the server—the proxy couldn't start up, right? Let's look at the log file—open /tmp/zabbix_proxy.log. Paying close attention we can find some interesting log records.

27982:20090729:131636.530 Cannot open database file "/tmp/zabbix_
proxy.db": No such file or directory
27982:20090729:131636.531 Creating database ...

Wonderful. Zabbix proxy can automatically create the required SQLite database and populate it.

We could also verify that Zabbix proxy is listening on the port it should be by running:

$ netstat -ntl | grep 10051

The output should confirm that everything is correct.

zabbix

Monitoring a host through a proxy

N ow that we have the proxy compiled, configured, and running, we have to inform Zabbix about it somehow. To do this, open Administration | DM in the frontend, then select Proxies in the first dropdown. No proxies are listed, so we have to create one—click the Create Proxy button and complete the form. Enter proxy in the Proxy name field.

The section below; Hosts, allows us to specify which hosts will be monitored by this proxy. To make one host monitored by proxy, mark Another Host in the Other Hosts list box and click on the << button.

zabbix

When you are done, click Save.

This server should now be monitored by the proxy, right? Again, not quite. As you might remember, we were setting the Zabbix server IP address in the agent configuration files so that agents would accept connections from these servers and send active check data to them. When we change a host to be monitored by a proxy, all connections will be made from the proxy, thus the proxy address must be added to the agent daemon's configuration file.

As root, edit /etc/zabbix/zabbix_agentd.conf on Another Host and change the Hostname line to have the Zabbix proxy IP address instead of the Zabbix server IP address. Save the file, then restart the agent daemon. Now this agent will send all data to the proxy only, no connection will be made to or from Zabbix server.

If, on the other hand, you have the proxy running on the same host as Zabbix server, agent will not notice that it is now queried by the proxy, not by the server, because the source IP would not have changed. The agent is still requesting and sending active checks to port 10051, Zabbix server port. To change this for the proxy port, edit /etc/zabbix/zabbix_agentd.conf on Another Host as root and change the ServerPort line to read:

ServerPort=10052

Don't forget to remove the leading hash mark, then restart the agent daemon.

With the host monitored solely by the proxy, let's check whether there are any indications of that in the frontend. Open Configuration | Hosts, make sure Linux servers is selected in the Group dropdown and take a look at the Name column. As can be seen, Another Host is now prefixed by the proxy name and reads proxy:Another Host.

zabbix

Of course, we are not limited in our proxy name choice, we could have named it "Nothing to see here" for all that the Zabbix server cares.

But do we have to always go to Administration | DM whenever we want to make a host monitored by proxy? Click on Another Host in the Name column, and observe the available properties. There's one dropdown available, Monitored by proxy. Using this dropdown we can easily assign a host to be monitored by the chosen proxy (remembering to change server IP address in the agent daemon configuration file, if using active checks).

zabbix

Zabbix 1.8 Network Monitoring Monitor your network hardware, servers, and web performance effectively and efficiently
Published: March 2010
eBook Price: $26.99
Book Price: $44.99
See more
Select your format and quantity:

Proxy benefits

But how independent is a proxy really? What happens if the server goes down? If we look at /etc/zabbix/zabbix_proxy.conf, we can find a configuration parameter named ProxyOfflineBuffer. This parameter controls how long the proxy will keep data in it's local database (in hours) in case Zabbix server cannot be contacted. Being one hour by default, this means the Zabbix server can be down or unreachable for up to one hour, and we would not lose any data as long as it came back online before the hour was up. Of course, you can increase this period but make sure you have enough free disk space, especially if running on embedded hardware.

There's one caveat—a proxy is a data collector only. This means that proxy does not process events, send out alerts, or execute remote commands. You need to pass the data to the Zabbix server so that it can do all those things.

Thu s proxies can gather data while the Zabbix server is down or unreachable for some reason and buffer this data for the configured period. If the Zabbix server was down for two hours, we would still lose one hour worth of data.

Another benefit is performance gain. This is not something to ignore; as your Zabbix installation grows, the load on the database can bring it down quite easily, especially if you are not careful with monitored items and their interval choices. As the proxy deals with all the hard work of polling, keeping an eye on when it has to do checks and providing Zabbix server with a nice, prepared chunk of data to insert into the database, in some configurations adding a proxy can notably reduce the load on the Zabbix server. Still, the proxy has to grab the configuration from Zabbix server now and then, and this process isn't as lightweight. The server has to figure out whether the configuration for the systems the proxy is responsible for contain data that has changed since the last synchronization, and this can be fairly resource intensive. If the configuration is very unbalanced, it can even decrease server performance. For example, having a proxy that synchronizes the configuration very often but only monitors a few items would usually make little sense.

By default, proxies synchronize the configuration once per hour, and this period can be set in /etc/zabbix/zabbix_proxy.conf configuration file. Look for parameter named ConfigFrequency, which by default will look like this:

# ConfigFrequency=3600

This means that a Zabbix proxy can lag in configuration up to an hour, which might sound scary but once a production installation settles, the configuration usually doesn't change that often. While testing, you might wish to decrease this period, but in a stable production setup it is actually suggested to increase this value.

Talking about the benefits a proxy provides, we already discussed the third big one, the ability to have only one way connections, even when monitoring things that can't have Zabbix agent installed such as various network devices, printers, and UPSes. That can help with firewalls allowing connections only in one direction, as well as requiring only a single port to be open instead of dozens for various services and protocols.

Proxy reliability

With all the benefits that a proxy brings, how can one resist embracing it wherever possible – especially on remote locations? Well, first, a Zabbix proxy doesn't work on thin air, it needs a server. If all you have on that remote location is SNMP, IPMI, or other protocols capable devices that have no way to install a Zabbix proxy, you'll have to live without this useful daemon.

Even if there is a machine that you can use for a Zabbix proxy, problems with that host will impact the monitoring of all other devices. As a proxy goes down, neither normal, active, SNMP, nor any other check would be processed. So one really must have a reliable host on the other end. But even reliable hosts can have problems. As we now know, all connections are initiated from the proxy (configuration synchronization and data transfer including), which essentially means Zabbix server has no idea whether proxy is still there. Well, this is not quite true, as a Zabbix server actually can find out whether something really bad has happened with a proxy. Let's try to figure out how we can determine when a proxy was last seen reporting. Open Configuration | Hosts, make sure Linux servers are selected in the Group dropdown and click on Items next to A Test Host then click on Create Item. Fill in the following values:

  • Description: Enter Last proxy access
  • Type: Select Zabbix internal
  • Key: Enter zabbix[proxy,proxy,lastaccess]
  • Units: Enter unixtime
  • Keep history: Enter 7

When you are done, click Save. Notice how we used special unit type—unixtime. Now what would it do? To find out, navigate to Monitoring | Latest data, make sure A Test Host is selected in the Host dropdown. Expand the filter and enter proxy there, then click the Filter button. Look at the way data is presented here—we can see very nicely in a human-readable form when the proxy last contacted server.

zabbix

So this item will be recording the time when proxy last contacted Zabbix server. That's great, but hardly enough to notice problems in everyday routine—we already know quite well that a trigger is absolutely needed. Here, already familiar fuzzytime() function comes to rescue. Navigate to Configuration | Hosts, select Triggers from the first dropdown, then click the Create Trigger button.

Let's say we have some fairly loaded and critical proxy—we would like to know when three minutes have passed without the proxy communicating back. In such a case, a trigger expression like this could be used:

{host:zabbix[proxy,proxy,lastaccess].fuzzytime(180)}=0

That's very nice, but if we recall when the proxy connects to the server, there were two cases—it either synchronized the configuration, or sent data. What if, for some reason, all occurrences of both these events are further apart than these three minutes? Nothing to fear. Zabbix proxy is a witty beast, and it also has a heartbeat process, which reports back to the server at regular intervals. Even better, this timing is configurable. Again, take a look at /etc/zabbix/zabbix_proxy.conf, this time looking for the HeartbeatFrequency variable, which by default looks like this:

# HeartbeatFrequency=60

Being specified in seconds, this value means that every 60 seconds the proxy will report back to the server, even if there are no new values to send, thus checking on proxy. The lastaccess item is quite a reliable way to figure out when a proxy is most likely down or at least inaccessible, even if it should not be sending data for a longer period of time. As usual, the heartbeat interval can be easily customized in the said configuration file; remember to uncomment the line after changing its value.

So for our trigger, fill in the following values:

  • Name: Proxy $2 not connected for 3 minutes
  • Expression: {A Test Host:zabbix[proxy,proxy,lastaccess]. fuzzytime(180)}=0

When done, click Save.

Remember, we talked about how you need a reliable server to run a Zabbix proxy on? There's also another option, which is more like turn-key solution. You could set up an appliance-like system, some simple, low spec device, that would explicitly run Zabbix proxy. Unless there's a large volume of data being processed, it could be really simple and use an automatically created SQLite database, thus plugging a cheap headless proxy device into some remote network could provide you with nice monitoring access.

Tweaking proxy configuration

While many configuration parameters for a proxy are the same as for the server (the pollers to start, port to listen on, and so on), and some are same as for the agent daemon (hostname), there are some proxy specific parameters. Knowing about these can be helpful when diagnosing a proxy-related problem, or when the proxy must be deployed in a specific environment.

Option

Description

ProxyLocalBuffer

Proxy will keep data in the local database for this many hours. By default all data that is synchronized to Zabbix server is removed.

ProxyOfflineBuffer

Proxy will keep data for this many hours if Zabbix server is unavailable. By default, data older than one hour is discarded.

HeartbeatFrequency

By default Zabbix proxy sends a heartbeat message to the Zabbix server every minute. This parameter allows to customize that.

ConfigFrequency

By default Zabbix proxy retrieves new configuration from server once an hour. As discussed above, you might want to increase this for large, fairly static setups, or maybe decrease for smaller, more dynamic installations.

DataSenderFrequency

This parameter specifies how often proxy pushes collected data to Zabbix server. By default it's one second. As all the trigger and alert processing is done by server, it is suggested to keep this value low.

Summary

We have now learned one useful method that will help us with monitoring remote locations that can't be reached directly. Let's rehash the main benefits that a Zabbix proxy provides:

  • Only connections from the Zabbix proxy to the Zabbix server on a single port are made, thus allowing us to monitor devices behind a firewall or devices that are inaccessible because of network configuration
  • Zabbix server is offloaded from keeping track of checks and actually performing them, thus increasing performance
  • Local buffering on the proxy allows it to continue gathering data while the Zabbix server is unavailable, transmitting it all when connectivity problems are resolved

With the listed benefits and being easy to install, a proxy is a very tempting part of the monitoring solution, but it is advisable to keep track of it by using the lastaccess Zabbix internal item.

If you have read this article you may be interested to view :

Zabbix 1.8 Network Monitoring Monitor your network hardware, servers, and web performance effectively and efficiently
Published: March 2010
eBook Price: $26.99
Book Price: $44.99
See more
Select your format and quantity:

About the Author :


Rihards Olups

Rihards Olups has over 10 years of experience in IT. He has had a chance to work with various systems, and most of that time has been spent with open source solutions. Exposure to Zabbix, one of the leading open source enterprise class monitoring solutions, was with the first public releases more than nine years ago, which has allowed to gain practical knowledge on the subject.

Previously employed by a government agency, Rihards was mostly involved in open source software deployments ranging from server to desktop grade software, with a big emphasis on Zabbix. More recently the author has joined Zabbix SIA, the company behind the software that this book is about, which has allowed him to gain even more experience with the subject.

Books From Packt

 

Building Telephony Systems with OpenSIPS 1.6
Building Telephony Systems with OpenSIPS 1.6

Funambol Mobile Open Source
Funambol Mobile Open Source

Beginning OpenVPN 2.0.9
Beginning OpenVPN 2.0.9

FreePBX 2.5 Powerful Telephony Solutions
FreePBX 2.5 Powerful Telephony Solutions

Cacti 0.8 Network Monitoring
Cacti 0.8 Network Monitoring

Spring Security 3 [RAW]
Spring Security 3 [RAW

CodeIgniter 1.7 professional development
CodeIgniter 1.7 professional development

Asterisk 1.6
Asterisk 1.6

 

Code Download and Errata
Packt Anytime, Anywhere
Register Books
Print Upgrades
eBook Downloads
Video Support
Contact Us
Awards Voting Nominations Previous Winners
Judges Open Source CMS Hall Of Fame CMS Most Promising Open Source Project Open Source E-Commerce Applications Open Source JavaScript Library Open Source Graphics Software
Resources
Open Source CMS Hall Of Fame CMS Most Promising Open Source Project Open Source E-Commerce Applications Open Source JavaScript Library Open Source Graphics Software