Ways to improve performance of your server in ModSecurity 2.5

Exclusive offer: get 50% off this eBook here
ModSecurity 2.5

ModSecurity 2.5 — Save 50%

Prevent web application hacking with this easy to use guide

$29.99    $15.00
by Magnus Mischel | November 2009 | Linux Servers Open Source

In this article by Magnus Mischel we will take a look at the performance of ModSecurity and what impact, if any, it has on the speed of your server and web application. We will also look at the ways in which you can increase the performance of your server by tweaking your configuration and writing more efficient rules.

A typical HTTP request

To get a better picture of the possible delay incurred when using a web application firewall, it helps to understand the anatomy of a typical HTTP request, and what processing time a typical web page download will incur. This will help us compare any added ModSecurity processing time to the overall time for the entire request.

When a user visits a web page, his browser first connects to the server and downloads the main resource requested by the user (for example, an .html file). It then parses the downloaded file to discover any additional files, such as images or scripts, that it must download to be able to render the page. Therefore, from the point of view of a web browser, the following sequence of events happens for each file:

  1. Connect to web server.
  2. Request required file.
  3. Wait for server to start serving file.
  4. Download file.

Each of these steps adds latency, or delay, to the request. A typical download time for a web page is on the order of hundreds of milliseconds per file for a home cable/DSL user. This can be slower or faster, depending on the speed of the connection and the geographical distance between the client and server.

If ModSecurity adds any delay to the page request, it will be to the server processing time, or in other words the time from when the client has connected to the server to when the last byte of content has been sent out to the client.

Another aspect that needs to be kept in mind is that ModSecurity will increase the memory usage of Apache. In what is probably the most common Apache configuration, known as "prefork", Apache starts one new child process for each active connection to the server. This means that the number of Apache instances increases and decreases depending on the number of client connections to the server.As the total memory usage of Apache depends on the number of child processes running and the memory usage of each child process, we should look at the way ModSecurity affects the memory usage of Apache.

A real-world performance test

In this section we will run a performance test on a real web server running Apache 2.2.8 on a Fedora Linux server (kernel 2.6.25). The server has an Intel Xeon 2.33 GHz dual-core processor and 2 GB of RAM.

We will start out benchmarking the server when it is running just Apache without having ModSecurity enabled. We will then run our tests with ModSecurity enabled but without any rules loaded. Finally, we will test ModSecurity with a ruleset loaded so that we can draw conclusions about how the performance is affected. The rules we will be using come supplied with ModSecurity and are called the "core ruleset".

The core ruleset

The ModSecurity core ruleset contains over 120 rules and is shipped with the default ModSecurity source distribution (it's contained in the rules sub-directory). This ruleset is designed to provide "out of the box" protection against some of the most common web attacks used today. Here are some of the things that the core ruleset protects against:

  • Suspicious HTTP requests (for example, missing User-Agent or Accept headers)
  • SQL injection
  • Cross-Site Scripting (XSS)
  • Remote code injection
  • File disclosure

We will examine these methods of attack, but for now, let's use the core ruleset and examine how enabling it impacts the performance of your web service.

Installing the core ruleset

To install the core ruleset, create a new sub-directory named modsec under your Apache conf directory (the location will vary depending on your distribution). Then copy all the .conf files from the rules sub-directory of the source distribution to the new modsec directory:

mkdir /etc/httpd/conf/modsec
cp/home/download/modsecurity-apache/rules/modsecurity_crs_*.conf /
etc/httpd/conf/modsec

Finally, enter the following line in your httpd.conf file and restart Apache to make it read the new rule files:


# Enable ModSecurity core ruleset
Include conf/modsecurity/*.conf

Putting the core rules in a separate directory makes it easy to disable them—all you have to do is comment out the above Include line in httpd.conf, restart Apache, and the rules will be disabled.

Making sure it works

The core ruleset contains a file named modsecurity_crs_10_config.conf. This file contains some of the basic configuration directives needed to turn on the rule engine and configure request and response body access. Since we have already configured these directives, we do not want this file to conflict with our existing configuration, and so we need to disable this. To do this, we simply need to rename the file so that it has a different extension as Apache only loads *.conf files with the Include directive we used above:

$ mv modsecurity_crs_10_config.conf modsecurity_crs_10_config.conf.
disabled

Once we have restarted Apache, we can test that the core ruleset is loaded by attempting to access an URL that it should block. For example, try surfing to http://yourserver/ftp.exe and you should get the error message Method Not Implemented, ensuring that the core rules are loaded.

Performance testing basics

So what effect does loading the core ruleset have on web application response time and how do we measure this? We could measure the response time for a single request with and without the core ruleset loaded, but this wouldn't have any statistical significance—it could happen that just as one of the requests was being processed, the server started to execute a processor-intensive scheduled task, causing a delayed response time.

The best way to compare the response times is to issue a large number of requests and look at the average time it takes for the server to respond.

An excellent tool—and the one we are going to use to benchmark the server in the following tests—is called httperf. Written by David Mosberger of Hewlett Packard Research Labs, httperf allows you to simulate high workloads against a web server and obtain statistical data on the performance of the server. You can obtain the program at http://www.hpl.hp.com/research/linux/httperf/ where you'll also find a useful manual page in the PDF file format and a link to the research paper published together with the first version of the tool.

Using httperf

We'll run httperf with the options --hog (use as many TCP ports as needed), --uri/index.html (request the static web page index.html) and we'll use --num-conn 1000 (initiate a total of 1000 connections). We will be varying the number of requests per second (specified using --rate) to see how the server responds under different workloads.

This is what the typical output from httperf looks like when run with the above options:

$ ./httperf --hog --server=bytelayer.com --uri /index.html --num-conn
1000
--rate 50
Total: connections 1000 requests 1000 replies 1000 test-duration
20.386 s
Connection rate: 49.1 conn/s (20.4 ms/conn, <=30 concurrent
connections)
Connection time [ms]: min 404.1 avg 408.2 max 591.3 median 404.5
stddev 16.9
Connection time [ms]: connect 102.3
Connection length [replies/conn]: 1.000
Request rate: 49.1 req/s (20.4 ms/req)
Request size [B]: 95.0
Reply rate [replies/s]: min 46.0 avg 49.0 max 50.0 stddev 2.0 (4
samples)
Reply time [ms]: response 103.1 transfer 202.9
Reply size [B]: header 244.0 content 19531.0 footer 0.0 (total
19775.0)
Reply status: 1xx=0 2xx=1000 3xx=0 4xx=0 5xx=0
CPU time [s]: user 2.37 system 17.14 (user 11.6% system 84.1% total
95.7%)
Net I/O: 951.9 KB/s (7.8*10^6 bps)
Errors: total 0 client-timo 0 socket-timo 0 connrefused 0 connreset 0
Errors: fd-unavail 0 addrunavail 0 ftab-full 0 other 0


The output shows us the number of TCP connections httperf initiated per second ("Connection rate"), the rate at which it requested files from the server ("Request rate"), and the actual reply rate that the server was able to provide ("Reply rate"). We also get statistics on the reply time—the "reply time – response" is the time taken from when the first byte of the request was sent to the server to when the first byte of the reply was received—in this case around 103 milliseconds. The transfer time is the time to receive the entire response from the server.

The page we will be requesting in this case, index.html, is 20 KB in size which is a pretty average size for an HTML document. httperf requests the page one time per connection and doesn't follow any links in the page to download additional embedded content or script files, so the number of such links in the page is of no relevance to our test.

Getting a baseline: Testing without ModSecurity

When running benchmarking tests like this one, it's always important to get a baseline result so that you know the performance of your server when the component you're measuring is not involved. In our case, we will run the tests against the server when ModSecurity is disabled. This will allow us to tell which impact, if any, running with ModSecurity enabled has on the server.

Response time

The following chart shows the response time, in milliseconds, of the server when it is running without ModSecurity. The number of requests per second is on the horizontal axis:

ModSecurity 2.5

As we can see, the server consistently delivers response times of around 300 milliseconds until we reach about 75 requests per second. Above this, the response time starts increasing, and at around 500 requests per second the response time is almost a second per request. This data is what we will use for comparison purposes when looking at the response time of the server after we enable ModSecurity.

Memory usage

Finding the memory usage on a Linux system can be quite tricky. Simply running the Linux top utility and looking at the amount of free memory doesn't quite cut it, and the reason is that Linux tries to use almost all free memory as a disk cache. So even on a system with several gigabytes of memory and no memory-hungry processes, you might see a free memory count of only 50 MB or so.

Another problem is that Apache uses many child processes, and to accurately measure the memory usage of Apache we need to sum the memory usage of each child process. What we need is a way to measure the memory usage of all the Apache child processes so that we can see how much memory the web server truly uses.

To solve this, here is a small shell script that I have written that runs the ps command to find all the Apache processes. It then passes the PID of each Apache process to pmap to find the memory usage, and finally uses awk to extract the memory usage (in KB) for summation. The result is that the memory usage of Apache is printed to the terminal.

The actual shell command is only one long line, but I've put it into a file called apache_mem.sh to make it easier to use:

#!/bin/sh
# apache_mem.sh
# Calculate the Apache memory usage
ps -ef | grep httpd | grep ^apache | awk '{ print $2 }' |
xargs pmap -x | grep 'total kB' | awk '{ print $3 }' |
awk '{ sum += $1 } END { print sum }'

Now, let's use this script to look at the memory usage of all of the Apache processes while we are running our performance test. The following graph shows the memory usage of Apache as the number of requests per second increases:

ModSecurity 2.5

Apache starts out consuming about 300 MB of memory. Memory usage grows steadily and at about 150 requests per second it starts climbing more rapidly.

At 500 requests per second, the memory usage is over 2.4 GB—more than the amount of physical RAM of the server. The fact that this is possible is because of the virtual memory architecture that Linux (and all modern operating systems) use. When there is no more physical RAM available, the kernel starts swapping memory pages out to disk, which allows it to continue operating. However, since reading and writing to a hard drive is much slower than to memory, this starts slowing down the server significantly, as evidenced by the increase in response time seen in the previous graph.

CPU usage

In both of the tests above, the server's CPU usage was consistently around 1 to 2%, no matter what the request rate was. You might have expected a graph of CPU usage in the previous and subsequent tests, but while I measured the CPU usage in each test, it turned out to run at this low utilization rate for all tests, so a graph would not be very useful. Suffice it to say that in these tests, CPU usage was not a factor.

ModSecurity without any loaded rules

Now, let's enable ModSecurity—but without loading any rules—and see what happens to the response time and memory usage. Both SecRequestBodyAccess and SecResponseBodyAccess were set to On, so if there is any performance penalty associated with buffering requests and responses, we should see this now that we are running ModSecurity without any rules.

The following graph shows the response time of Apache with ModSecurity enabled:

ModSecurity 2.5

We can see that the response time graph looks very similar to the response time graph we got when ModSecurity was disabled. The response time starts increasing at around 75 requests per second, and once we pass 350 requests per second, things really start going downhill.

The memory usage graph is also almost identical to the previous one:

ModSecurity 2.5

Apache uses around 1.3 MB extra per child process when ModSecurity is loaded, which equals a total increase of memory usage of 26 MB for this particular setup. Compared to the total amount of memory Apache uses when the server is idle (around 300 MB) this equals an increase of about 10%.

Mod Security with the core ruleset loaded

Now for the really interesting test we'll run httperf against ModSecurity with the core ruleset loaded and look at what that does to the response time and memory usage.

Response time

The following graph shows the server response time with the core ruleset loaded:

ModSecurity 2.5

At first, the response time is around 340 ms, which is about 35 ms slower than in previous tests. Once the request rate gets above 50, the server response time starts deteriorating. As the request rates grows, the response time gets worse and worse, reaching a full 5 seconds at 100 requests per second. I have capped the graph at 100 requests per second, as the server performance has already deteriorated enough at this point to allow us to see the trend.

We see that the point at which memory usage starts increasing has gone down from 75 to 50 requests per second now that we have enabled the core ruleset. This equals a reduction in the maximum number of requests per second the server can handle of 33%.

ModSecurity 2.5 Prevent web application hacking with this easy to use guide
Published: November 2009
eBook Price: $29.99
Book Price: $49.99
See more
Select your format and quantity:

Memory usage

What could be causing this deterioration in response time? Let's take a look at the memory usage of Apache and see if we can find any clues:

ModSecurity 2.5

Aha! We see that once we hit 50 requests per second, the memory usage goes up dramatically. The server only has 2 GB of memory, so it's a pretty good bet that the increase in memory usage and subsequent swapping of memory pages to and from the hard drive is what causes the server performance to deteriorate.

For comparison purposes, take a look at these graphs, which show the response rate and memory usage for ModSecurity with no rules (dotted line) and ModSecurity with the core ruleset (solid line):

ModSecurity 2.5

The memory usage with the core ruleset loaded is represented by the solid line in the following graph:

ModSecurity 2.5

The conclusion we can draw from this is that once the request rate goes over a certain threshold, the memory usage grows to the extent that the kernel has to start swapping data to disk. Once this happens, the response time grows larger and larger.

We see that under 50 requests per second the response times are virtually identical, indicating that ModSecurity does not incur any significant performance penalty as long as there is a sufficient amount of free memory available. This is important, because it shows that the underlying limit being encountered is the amount of free memory, and this could have happened just as easily without ModSecurity enabled. ModSecurity just lowers the threshold at which this happens.

Finding the bottleneck

Is it possible that the deterioration in response time seen in the previous graphs is caused by a ModSecurity configuration setting? Two likely candidates would be request and response body buffering. Both of these settings, when set to On, cause ModSecurity to allocate extra memory to hold the buffered request and response bodies.

Let's set both RequestBodyAccess and ResponseBodyAccess to Off and run the same tests again and see if there's any difference.

ModSecurity 2.5

When buffering is disabled, the response time doesn't start increasing until we get above 75 requests per second, indicating that turning off buffering indeed improved performance—the server is able to handle 25 extra requests per second before performance starts going downhill.

Let's see what the memory usage looks like with buffering disabled:

ModSecurity 2.5

As expected, we see that with buffering turned off, memory usage doesn't start increasing significantly until we reach 75 requests per second, again showing that memory load is a crucial factor in the number of requests per second the server can handle.

Wrapping up core ruleset performance

We have seen that loading the core ruleset caused a decrease in the number of simultaneous connections the server could handle before being overwhelmed. CPU usage was not an issue at all, but the server was running a pretty fast dual-core processor, so the results may be different on systems with older hardware.

In the case of the limitation we ran into here, optimizing two things should enable us to squeeze a significant amount of extra performance out of the server:

  • Decreasing the memory usage of each Apache process
  • Adding extra RAM to the server

In the next section we will be looking at how to decrease the amount of memory Apache uses, and we will also be looking at ways to optimize your rules for efficiency.

Optimizing performance

The previous results indicate that you will likely not see any performance degradation from using ModSecurity unless Apache starts consuming too much memory, or you are using a large number of rules and a slow system. What can you do if you do run into either the memory or processor becoming overloaded? This part of the article gives some practical advice to help you squeeze the best performance out of your ModSecurity setup.

Memory consumption

Adding extra RAM to your server may not be the first or easiest thing to do when you find that the web server processes use too much memory, and therefore it pays to know how to decrease the memory footprint of each Apache process.

Follow these tips and you may not have to add any extra memory:

  • Decrease Apache module usage

    The number of dynamic modules that Apache has to load has a direct impact on the memory footprint of each process. Therefore, tweak your httpd.conf file so that any modules which you don't require are disabled (by simply commenting out the LoadModule lines for the modules you don't need).

  • Limit the number of requests per Apache child process

    If you are using Apache in the prefork configuration (one child process per request) then you should set MaxRequestsPerChild so that each child's memory usage gets reset after a certain number of requests. Apache child processes tend to grow over time and seldom do they shrink in size unless restarted.

  • Reduce the number of ModSecurity rules

    The larger your ModSecurity ruleset, the more memory each Apache process will consume. In addition, a large ruleset takes a longer time to execute, so the performance savings are two-fold.

Bypassing inspection of static content

You can often achieve a performance gain in general by passing requests for static content such as images or binary files to a web server specialized for this task. If you don't want to install a light-weight web server especially to handle static content, you can configure ModSecurity to not inspect such fi les by using a rule similar to the following:

SecRule REQUEST_FILENAME ".(?:jpe?g|gif|png|js|exe)$"
"phase:1,allow"

The above rule immediately allows access to files ending in .jpg, .jpeg, .gif, .png, .js, and .exe. It's easy enough to add your own extensions for any additional static content you may have on your site with other extensions.

Using @pm and @pmFromFile

We saw in that using the @pm and @pmFromFile operators can be quicker than a standard regex matching attempt. But just how much quicker are these operators?

As an example, imagine that you would like to block requests with a particular referrer header. A list of "banned" referrers could be several hundred or even thousand lines long, so it's important to know which method to use and the differences in speed between them.

To investigate the difference in speed between regex matching and the @pm and @pmFromFile operators, let's look at how ModSecurity performs when we throw a list of 500 phrases at it. For this test, let's use a dictionary file and our favorite programming language to generate a list of 500 random domain names to block. The list starts out like this, just to give you an idea of what we're working with:

gaddersprossies.org
bastefilagree.com
thicksetflurry.net
shaitansshiners.org
colludesfeminise.com
luffingsall.com
oversewnprinker.net
metereddebonair.com
sparingcricking.com
...

We'll save this file as DomainNames.txt and upload it to our server running Apache and ModSecurity. We can then take the list of domain names and modify them so that each line looks as follows:

SecRule REQUEST_HEADERS:Referer "gaddersprossies.org" deny
SecRule REQUEST_HEADERS:Referer "bastefilagree.com" deny
SecRule REQUEST_HEADERS:Referer "thicksetflurry.net" deny
SecRule REQUEST_HEADERS:Referer "shaitansshiners.org" deny
SecRule REQUEST_HEADERS:Referer "colludesfeminise.com" deny
SecRule REQUEST_HEADERS:Referer "luffingsall.com" deny
SecRule REQUEST_HEADERS:Referer "oversewnprinker.net" deny
SecRule REQUEST_HEADERS:Referer "metereddebonair.com" deny
SecRule REQUEST_HEADERS:Referer "sparingcricking.com" deny
...

As you can see, this file now contains ModSecurity rules to block any referrer containing one of the specifi ed domain names. We'll save this file as DomainNamesRegex.conf and place it in the conf.d subdirectory of the Apache root, causing the rules to be loaded after an Apache restart. We'll also set the debug log level to 4, which causes ModSecurity to log timestamps at the beginning and end of each phase to the debug log:

These are the results when accessing a file on the server with the regex rules in place:

[rid#b8a763b8][/][4] Time #1: 258
...
[rid#b8a763b8][/][4] Time #2: 13616

The times are given in microseconds, so the total time spent processing the regex rules was 13616–258 = 13348 microseconds, or about 13.3 milliseconds.

Now let's see how much time the @pmFromFile operator takes:

[rid#b8a69bf0][/][4] Time #1: 414
...
[rid#b8a69bf0][/][4] Time #2: 485

This time the total time spent was 485–414 = 71 microseconds, or about 0.07 milliseconds. This data indicates that @pmFromFile is approximately 200 times faster than using a regular expression. It's pretty clear that you should prefer using @pm and @pmFromFile for large lists of phrases that need to be matched.

Logging

Enabling debug and audit logging will cause ModSecurity to write log entries to the respective log file. Especially with debug logging at a high level (for example, 9), you will incur a performance penalty since each request will generate many lines of log data. Therefore, you should not enable debug logging unless you are testing out new rules or debugging a problem.

Writing regular expressions for best performance

There are certain things you can do when writing your regular expressions that will ensure that you achieve maximum performance. What follows are a few guidelines on things you should and shouldn't do when writing regexes.

Use non-capturing parentheses wherever possible

Non-capturing parentheses, as we have mentioned earlier, are this awkward-looking construct:

(?: )

They perform the same function as regular parentheses, with the difference that the non-capturing ones do not capture backreferences. This means the regular expression engine doesn't need to keep track of backreferences or allocate memory for them, which saves some processing time as well as memory.

If you look at the core ruleset, you'll notice that many of the rules in it contain these parentheses, which shows you that if performance is something you have in mind then you should be using them, too.

Use one regular expression whenever possible

It is usually more efficient to use a single regular expression instead of a lot of smaller ones. So if for example you wanted to match a filename extension, the following rule would be faster than using one rule for each extension:

SecRule REQUEST_FILENAME ".(?:exe|bat|pif)" deny

You'll have to weigh writing one-line, complex rules against the readability of your ruleset. For a small amount of rules in the ruleset, the difference in speed won't matter, and you should prefer to use simple, readable rules. If your ruleset starts growing into the hundreds of rules, you may want to consider using the above technique.

Summary

In this article we looked at the performance of ModSecurity. The results when benchmarking ModSecurity indicate that the additional latency due to CPU usage is usually low. Apache's memory usage increases when ModSecurity is enabled and is using the approximately 120 rules in the core ruleset, and we have seen that this leads to a decrease in the number of simultaneous connections that the server can successfully handle due to increased memory usage.

In most cases, enabling ModSecurity should not slow down your server unless you are getting a lot of concurrent requests. If you do experience a slow-down (or are able to measure a significant one using a benchmarking tool such as httperf) then it is important to find out the underlying cause.

If the problem is that Apache uses too much memory then you need to either configure it (and ModSecurity) to use less memory, add more RAM, or both. If the CPU usage goes up and you find that this is caused by ModSecurity then implement the tips found in the last section of this article and also consider trimming the number of rules in your ruleset.

ModSecurity 2.5 Prevent web application hacking with this easy to use guide
Published: November 2009
eBook Price: $29.99
Book Price: $49.99
See more
Select your format and quantity:

About the Author :


Magnus Mischel

Computer security expert Magnus Mischel is the founder and director of Mischel Internet Security, whose product TrojanHunter helps protect computers from malware. He currently lives in London, and when he isn't writing books or managing the company, he enjoys playing a game of chess at the Metropolitan Chess Club. He holds an MSc in Computer Science and Engineering from Linköping University, Sweden.

Books From Packt


jQuery UI 1.7: The User Interface Library for jQuery
jQuery UI 1.7: The User Interface Library for jQuery

Matplotlib for Python Developers
Matplotlib for Python Developers

Apache Geronimo 2.1: Quick Reference
Apache Geronimo 2.1: Quick Reference

Linux Email
Linux Email

FreePBX 2.5 Powerful Telephony Solutions
FreePBX 2.5 Powerful Telephony Solutions

ADempiere 3.4 ERP Solutions
ADempiere 3.4 ERP Solutions

Moodle 1.9 Math
Moodle 1.9 Math

Asterisk 1.4 – the Professional’s Guide
Asterisk 1.4 – the Professional’s Guide


Your rating: None Average: 4.7 (3 votes)
Great article! by
Thank you very much for this great article. Never though that mod_security could consume such memory until I realized that it was increasing my Apache processes by 70Mb !

Post new comment

CAPTCHA
This question is for testing whether you are a human visitor and to prevent automated spam submissions.
Q
U
T
p
V
M
Enter the code without spaces and pay attention to upper/lower case.
Code Download and Errata
Packt Anytime, Anywhere
Register Books
Print Upgrades
eBook Downloads
Video Support
Contact Us
Awards Voting Nominations Previous Winners
Judges Open Source CMS Hall Of Fame CMS Most Promising Open Source Project Open Source E-Commerce Applications Open Source JavaScript Library Open Source Graphics Software
Resources
Open Source CMS Hall Of Fame CMS Most Promising Open Source Project Open Source E-Commerce Applications Open Source JavaScript Library Open Source Graphics Software