The Core HTTP Module in Nginx

Exclusive offer: get 50% off this eBook here
Nginx 1 Web Server Implementation Cookbook

Nginx 1 Web Server Implementation Cookbook — Save 50%

Over 100 recipes to master using the Nginx HTTP server and reverse proxy

$23.99    $12.00
by Dipankar Sarkar | July 2011 | Open Source

Nginx is an open source high-performance web server, which has gained quite some popularity recently. Due to its modular architecture and small footprint, it has been the default choice for a lot of smaller Web 2.0 companies to be used as a load-balancing proxy server. It supports most of the existing backend web protocols such as FCGI, WSGI, and SCGI.

In this article by Dipankar Sarkar, author of Nginx 1 Web Server Implementation Cookbook, we will cover:

  • Setting up the number of worker processes correctly
  • Increasing the size of uploaded files
  • Using dynamic SSI for simple sites
  • Adding content before and after a particular page
  • Enabling auto indexing of a directory
  • Serving any random web page from a directory
  • Serving cookies for identifying and logging users
  • Re-encoding the response to another encoding
  • Enabling Gzip compression on some content types
  • Setting up 404 and other error pages

 

Nginx 1 Web Server Implementation Cookbook

Nginx 1 Web Server Implementation Cookbook

Over 100 recipes to master using the Nginx HTTP server and reverse proxy

        Read more about this book      

(For more resources on Nginx, see here.)

Setting up the number of worker processes correctly

Nginx like any other UNIX-based server software, works by spawning multiple processes and allows the configuration of various parameters around them as well. One of the basic configurations is the number of worker processes spawned! It is by far one of the first things that one has to configure in Nginx.

How to do it...

This particular configuration can be found at the top of the sample configuration file nginx.conf:

user www www;
worker_processes 5;
error_log logs/error.log;
pid logs/nginx.pid;
worker_rlimit_nofile 8192;
events {
worker_connections 4096;
}

In the preceding configuration, we can see how the various process configurations work. You first set the UNIX user under which the process runs, then you can set the number of worker processes that Nginx needs to spawn, after that we have some file locations where the errors are logged and the PIDs (process IDs) are saved.

How it works...

By default, worker_processes is set at 2. It is a crucial setting in a high performance environment as Nginx uses it for the following reasons:

  • It uses SMP, which allows you to efficiently use multi-cores or multi-processors systems very efficiently and have a definite performance gain.
  • It increases the number of processes decreases latency as workers get blocked on disk I/O.
  • It limits the number of connections per process when any of the various supported event types are used. A worker process cannot have more connections than specified by the worker_connections directive.

There's more...

It is recommended that you set worker_processes as the number of cores available on your server. If you know the values of worker_processes and worker_connections, one can easily calculate the maximum number of connections that Nginx can handle in the current setup.

Maximum clients = worker_processes * worker_connections

 

Increasing the size of uploaded files

Usually when you are running a site where the user uploads a lot of files, you will see that when they upload a file which is more than 1MB in size you get an Nginx error stating, "Request entity too Large" (413), as shown in the following screenshot. We will look at how Nginx can be configured to handle larger uploads.

Nginx 1 Web Server Implementation Cookbook

How to do it...

This is controlled by one simple part of the Nginx configuration. You can simply paste this in the server part of the Nginx configuration:

client_max_body_size 100M; # M stands for megabytes

This preceding configuration will allow you to upload a 100 megabyte file. Anything more than that, and you will receive a 413. You can set this to any value which is less than the available disk space to Nginx, which is primarily because Nginx downloads the file to a temporary location before forwarding it to the backend application.

There's more...

Nginx also lets us control other factors related to people uploading files on the web application, like timeouts in case the client has a slow connection. A slow client can keep one of your application threads busy and thus potentially slow down your application. This is a problem that is experienced on all the heavy multimedia user-driven sites, where the consumer uploads all kinds of rich data such as images, documents, videos, and so on. So it is sensible to set low timeouts.

client_body_timeout 60; # parameter in seconds
client_body_buffer_size 8k;
client_header_timeout 60; # parameter in seconds
client_header_buffer_size 1k;

So, here the first two settings help you control the timeout when the body is not received at one read-step (basically, if the server is queried and no response comes back). Similarly, you can set the timeout for the HTTP header as well. The following table lists out the various directives and limits you can set around client uploading.

Nginx 1 Web Server Implementation Cookbook

 

Using dynamic SSI for simple sites

With the advent of modern feature-full web servers, most of them have Server-Side Includes (SSI) built in. Nginx provides easy SSI support which can let you do pretty much all basic web stuff.

How to do it...

Let's take a simple example and start understanding what one can achieve with it.

  1. Add the following code to the nginx.conf file:

    server {
    .....
    location / {
    ssi on;
    root /var/www/www.example1.com;
    }
    }

  2. Add the following code to the index.html file:

    <html>
    <body>
    <!--# block name="header_default" -->
    the header testing
    <!--# endblock -->
    <!--# include file="header.html" stub="header_default" -->
    <!--# echo var="name" default="no" -->
    <!--# include file="footer.html"-->
    </body>
    </html>

  3. Add the following code to the header.html file:
    <h2>Simple header</h2>
  4. Add the following code to the footer.html file:
    <h2>Simple footer</h2>

How it works...

This is a simple example where we can see that you can simply include some partials in the larger page, and in addition to that you can create block as well within the page. So the <block> directive allows you to create silent blocks that can be included later, while the <include> directive can be used to include HTML partials from other files, or even URL end points. The <echo> directive is used to output certain variables from within the Nginx context.

There's more...

You can utilize this feature for all kinds of interesting setups where:

  • You are serving different blocks of HTML for different browsers types
  • You want to optimize and speed up certain common blocks of the sites
  • You want to build a simple site with template inheritance without installing any other scripting language

 

Adding content before and after a particular page

Today, in most of the sites that we visit, the webpage structure is formally divided into a set of boxes. Usually, all sites have a static header and a footer block. Here, in this following page you can see the YUI builder generating the basic framework of such a page.

In such a scenario, Nginx has a really useful way of adding content before and after it serves a certain page. This will potentially allow you to separate the various blocks and optimize their performance individually, as well.

Let's have a look at an example page:

Nginx 1 Web Server Implementation Cookbook

So here we want to insert the header block before the content, and then append the footer block:

Nginx 1 Web Server Implementation Cookbook

How to do it…

The sample configuration for this particular page would look like this:

server {
listen 80;
server_name www.example1.com;
location / {
add_before_body /red_block
add_after_body /blue_block;
...
}
location /red_block/ {
...
}
location /blue_block/ {
....
}
}

This can act as a performance enhancer by allowing you to load CSS based upon the browser only. There can be cases where you want to introduce something into the header or the footer on short notice, without modifying your backend application. This provides an easy fix for those situations.

This module is not installed by default and it is necessary to enable it when building Nginx.
./configure –with-http_addition_module

 

Enabling auto indexing of a directory

Nginx has an inbuilt auto-indexing module. Any request where the index file is not found will route to this module. This is similar to the directory listing that Apache displays.

How to do it...

Here is the example of one such Nginx directory listing. It is pretty useful when you want to share some files over your local network. To start auto index on any directory all you need to do is to carry out the following example and place it in the server section of the Nginx configuration file:

server {
location 80;
server_name www.example1.com;
location / {
root /var/www/test;
autoindex on;
}
}

How it works...

This will simply enable auto indexing when the user types in http://www.example1.com. You can also control some other things in the listings in this way:

autoindex_exact_size off;

This will turn off the exact file size listing and will only show the estimated sizes. This can be useful when you are worried about file privacy issues.

autoindex_localtime on;

This will represent the timestamps on the files as your local server time (it is GMT by default):

Nginx 1 Web Server Implementation Cookbook

This image displays a sample index auto-generated by Nginx using the preceding configuration. You can see the filenames, timestamp, and the file sizes as the three data columns.

 

Nginx 1 Web Server Implementation Cookbook Over 100 recipes to master using the Nginx HTTP server and reverse proxy
Published: May 2011
eBook Price: $23.99
Book Price: $39.99
See more
Select your format and quantity:

 

        Read more about this book      

(For more resources on Nginx, see here.)

Serving any random web page from a directory

There has been a recent trend for a lot of sites to test out their new pages based upon the A/B methodology. You can explore more about its history and the various companies that have adopted this successfully as a part of their development process at http://en.wikipedia.org/wiki/A/B_testing. In this practice, you have a set of pages and some metric (such as number of registrations, or the number of clicks on a particular element). Then you go about getting people to randomly visit these pages and get data about their behavior on them. This lets you iteratively improve the page and the elements on them.

Nginx 1 Web Server Implementation Cookbook

Nginx has something that will let you to run your own A-B test without writing any code at all. It allows you to randomly select any web page from a directory and display it.

How to do it...

Let's have a look at a sample configuration which needs to be placed within the HTTP section of the Nginx configuration:

server {
listen 80;
server_name www.example1.com;
location / {
root /var/www/www.example1.com/test_index;
random_index on;
}
}

How it works...

Let's assume that you have some files in the /var/www/www.example1.com/test_index directory. When you turn on the random index it will scan the directory and then send a randomly picked file instead of the default index.html. The only exceptions are plain files. Whole filenames which start with a dot will not be part of the site of files to be picked from.

So here are two sample test pages, with slightly differing headers. Notice that the URLs are the same. So it will let you determine if the end user is clicking through more with the red link or the blue link using pure statistical methods:

Nginx 1 Web Server Implementation Cookbook

The preceding screenshot displays A.html on opening the site. There is equal probability of opening both the pages, much like the tossing of a coin and getting heads or tails.

Nginx 1 Web Server Implementation Cookbook

So, using the A-B testing as an example, you can set an A.html and a B.html, which would be served to the user randomly. It would allow you to easily measure a lot of interesting client behavior by simply analyzing the Nginx access logs.

 

Serving cookies for identifying and logging users

Nginx has a useful functionality of serving cookies for identifying users. This is very useful in tracking anonymous user behavior in case a website does not want to employ external analytics software. This module is compatible with the mod_uid module in Apache2, which provides a similar functionality.

How to do it…

Here is a sample configuration for this module. This goes in the server section of the configuration:

userid on;
userid_name uid;
userid_domain example1.com;
userid_path /;
userid_expires 365d;
userid_p3p 'policyref="/w3c/p3p.xml", CP="CUR ADM OUR NOR STA

How it works...

Now let's see and understand what the various directives are about. The first userid directive enables this module; the second assigns a name to the cookie which is going to be written on the client side. The next three directives are the standard cookie information that is needed (the primary domain, the path, and the time of expiry). The last directive enables the browser to understand the privacy practices that the website follows. This is done by using the P3P protocol which allows websites to declare their intended usage that they collect about the user. It is basically an XML file that allows you to programmatically display your privacy policy. The following code is a simple example configuration of how you can define a policy where the data is removed after 4 months:

<META xmlns="http://www.w3.org/2002/01/P3Pv1">
<POLICY-REFERENCES>
<EXPIRY max-age="10000000"/><!-- about four months -->
</POLICY-REFERENCES>
</META>

This XML put on the server will objectively define the privacy policies of the site to the incoming bots or users.

There's more...

On enabling this module, some variables are available in the Nginx configuration which allow you do fairly interesting things. You have access to some variables in the configuration contest, like $uid_got,$uid_set.

These can be used for writing interesting rewrite rules. A simple application using these variables is to log the users coming on your site and then determining the user bounce rates on your website by parsing the logs.

 

Re-encoding the response to another encoding

File encoding is a major issue on most websites, a lot of time the database (MySQL in most cases) is configured to run using the Latin-1 encoding instead of the UTF-8 encoding that is the prevalent standard. Nginx provides an easy solution for changing your web page encoding on-the-fly, so that your users do not end up with garbled characters on your website.

How to do it...

All you need to do is to place this in the server section of your Nginx configuration:

charset windows-1251;
source_charset koi8-r;

How it works...

This basically defines the fact that the source character set is koi8-r. If the encoding is different from the charset character set, then re-encoding is carried out. In case your original response already has a "Content-Type" header present then you will need to use the following to override and do the re-encoding:

overrride_charset on;

There's more...

You can also decide how the re-encoding happens by defining a character mapping. A simple example is the following:

charset_map koi8-r windows-1251 {
C0 FE ; # small yu
C1 E0 ; # small a
C2 E1 ; # small b
C3 F6 ; # small ts
# ...
}

Nginx lets you do these neat little things that can make your site more accessible and usable for the end-user.

 

Enabling Gzip compression on some content types

As the Web has evolved, we have had improvements in web server and browser technologies. In recent times, with the booming consumer Internet market, the web application has had to become faster.

Compression techniques, which were already present, have come of age and now most sites enable a fairly high degree of compression on the pages they serve. Nginx being state of the art, has Gzip compression and allows a whole lot of options on how to go about it.

How to do it...

You will need to modify your Nginx configuration file and add the following directives:

http {
gzip on;
gzip_min_length 1000;
gzip_comp_level 6;
gzip_proxied expired no-cache no-store private auth;
gzip_types text/plain application/xml;
gzip_disable "MSIE [1-6]\.";
server {
....
}
}

How it works...

This sample configuration allows you to turn on the Gzip compression of the outgoing page for all pages which are over 1000 bytes. This limit is set because compression technology performance degrades as the page size becomes smaller. You can then set the various MIME types for which the compression should occur; this particular example will compress only plain text files and XML files.

Older browsers are not the best when it comes to utilizing this, and you can disable Gzip depending on the browser type. One of the most interesting settings is the level of compression where you need to make a choice between the amount of CPU that you want to spend on compressing and serving the pages (the higher this number, more of your CPU time will go towards compressing and sending pages). It is recommended to follow a middle path on this particular setting; the client also spends more CPU time decompressing the page if you set this. A sensible setting of this value would be six.

There's more...

For proxy requests, gzip_proxied actually allows or disallows the compression of the response of the proxy request based on the request and the response. You can use the following parameters:

Nginx 1 Web Server Implementation Cookbook

So in the preceding example (expired no-cache no-store private auth) it is clear that the compression is enabled when the Expires header prevents caching, when the Cache-Control contains no-cache, no-store, or private, and when there is an Authorization header present. This allows tremendous control on how the compression is delivered to the client's browser.

 

Setting up 404 and other error pages

All web applications have errors and missing pages, and Nginx has easy methods of ensuring that the end user has a good experience when the application does not respond correctly. It successfully handles all the HTTP errors with default pages, which can gracefully notify the users that something has gone wrong.

How to do it...

Nginx allows you to do pretty interesting things with error pages. Following are some example configurations which can be placed within the HTTP or server section.

We are also going to define a named location using the "@" prefix after location. These locations are not used during the normal processing of any request and are intended to only process internally redirected requests.

location @fallback (
proxy_pass http://backend;
)
error_page 404 /404.html;
error_page 502 503 504 /50x.html;
error_page 403 http://example1.com/forbidden.html;
error_page 404 = @fallback;
error_page 404 =200 /.empty.gif;

How it works...

The first example allows you to map a simple 404 page to a simple HTML. The next example allows the mapping of various application error codes to another generic application error HTML page. You can also map the error page to some other external site all together (http://example1.com/forbidden.html). The fourth example allows you to map the page to another location, defined as @fallback. The last example is interesting as it actually allows you to change the response code to a 200 (HTTP OK). This is useful in situations where you have excessive 404 pages on the site, and would prefer not sending a 404 back as reply, but a 200 with a very small GIF file in return.

You can utilize this very effectively to give the end user a better experience when they inadvertently reach dead ends and application errors on your site.

If you do not set these error pages correctly, you will get the default Nginx error pages which may not be useful to the user and may turn them away.

Nginx 1 Web Server Implementation Cookbook

Summary

In this article we took a look at the core HTTP module in Nginx.


Further resources on this subject:


Nginx 1 Web Server Implementation Cookbook Over 100 recipes to master using the Nginx HTTP server and reverse proxy
Published: May 2011
eBook Price: $23.99
Book Price: $39.99
See more
Select your format and quantity:

About the Author :


Dipankar Sarkar

Dipankar Sarkar is a web and mobile entrepreneur. He has a Bachelor’s degree in Computer Science and Engineering from Indian Institute of Technology, Delhi. He is a firm believer in the Open source movement and has participated in the Google Summer of Code, 2005-06 and 2006-07. He has conducted technical workshops for Windows mobile and Python at various technical meetups. He recently took part in the Startup Leadership Program, Delhi Chapter.

He has worked with Slideshare LLC, one of the world’s largest online presentation hosting and sharing services as an early engineering employee. He has since then worked with Mpower Mobile LLC, a mobile payment startup and Clickable LLC, a leading search engine marketing startup. He was a co-founder at Kwippy, which was one of the top micro-blogging sites. He is currently working in the social TV space and has co-founded Jaja.

He has previously authored “Nginx web server implementation cookbook” and this is his second book on Nginx. This book “Mastering Nginx” is a more structured approach to how one can learn and master Nginx, with practical examples and strategies.

Books From Packt


Nginx HTTP Server
Nginx HTTP Server

Squid Proxy Server 3.1: Beginner's Guide
Squid Proxy Server 3.1: Beginner's Guide

Plone 3.3 Site Administration
Plone 3.3 Site Administration

Hacking Vim 7.2
Hacking Vim 7.2

Cacti 0.8 Beginner's Guide
Cacti 0.8 Beginner's Guide

FreeSWITCH 1.0.6
FreeSWITCH 1.0.6

OpenVPN 2 Cookbook
OpenVPN 2 Cookbook

Tcl 8.5 Network Programming
Tcl 8.5 Network Programming


No votes yet

Post new comment

CAPTCHA
This question is for testing whether you are a human visitor and to prevent automated spam submissions.
5
B
W
5
F
r
Enter the code without spaces and pay attention to upper/lower case.
Code Download and Errata
Packt Anytime, Anywhere
Register Books
Print Upgrades
eBook Downloads
Video Support
Contact Us
Awards Voting Nominations Previous Winners
Judges Open Source CMS Hall Of Fame CMS Most Promising Open Source Project Open Source E-Commerce Applications Open Source JavaScript Library Open Source Graphics Software
Resources
Open Source CMS Hall Of Fame CMS Most Promising Open Source Project Open Source E-Commerce Applications Open Source JavaScript Library Open Source Graphics Software