|Read more about this book|
(For more resources on Magento, see here.)
The reader can benefit from the previous article on How to Overcome the Pitfalls of Magento.
Users really respond to speed.—Marissa Mayer, Google vice president of the research section and user experience.
We will explain why this quote is true throughout this article. Her key insight for the crowd at the Web 2.0 Summit is that "slow and steady doesn't win the race". Today people want fast and furious. Not convinced? Okay, let's have a look at some arguments:
- 500ms lost is to lose 20 percent of traffic for Google (this might be why there are only ten results per page in the research).
- Increased latency of 100ms costs 1 percent of sales for Amazon.
- Reducing by 25 percent the weight of a page is to win 25 percent of users in the medium term for Google.
- Losing 400ms is to have a 5-9 percent drop in addition to Yahoo!, an editorial site.
This is the era of milliseconds and terabytes, so we have to pay a big price if we can't keep up. This article will describe how to ensure the optimum performance of your Magento store.
Measuring/benchmarking your Magento with Siege, ab, Magento profiler, YSlow, Page Speed, GTmetrix, and WebPagetest
The very first task of any website's performance optimization is to know its pitfalls. In other words, know why it is taking too much time. Who are the culprits? Fortunately, we have some amicable friends who will guide us through. Let's list them:
- ab (ApacheBench): This is bundled with every Apache as a benchmarking utility.
- Siege: This is an open source stress/regression test and benchmark utility by Joe Dog.
- Magento profiler: This is a built-in Magento profiler.
- YSlow: This is a tool from Yahoo! We have been using it for years. This is in fact a firebug add-on.
- Page Speed: This is yet another firebug add-on from Google to analyze page performance on some common rules.
- GTmetrix: This is a cool online web application from which you can get both YSlow and Page Speed in the same place. Opera fanboys who don't like Firefox or Chrome might use it for YSlow and Page Speed here.
- WebPagetest: This is another online tool for benchmarking as GTmetrix. It also collects and shows screenshots with the reports.
Okay, we are introduced to our new friends. In this recipe, we will work with them and find the pitfalls of our Magento store.
Before starting the work, we have to make sure that every required tool is in place. Let's check it.
ab: This Apache benchmarking tool is bundled with every Apache installation. If you are on a Linux-based distribution, you can give it a go by issuing the following command in the terminal:
Siege: We will use this tool in the same box as our server. So make sure you have it installed. You can see it by typing this command (note that the option is capital V):
If it's installed, you should see the installed version information of Siege. If it's not, you can install it with the following command in any Debian-based Distro:
sudo apt-get install siege
You can also install it from source. To do so, grab the latest source from here: ftp://ftp.joedog.org/pub/siege/siege-latest.tar.gz, then issue the following steps sequentially:
# go the location where you downloaded siege
tar xvzf siege-latest.tar.gz
# go to the siege folder. You should read it with something like
If you are on a Windows-based box, you would find it as:
Magento Profile: This is also a built-in tool with Magento.
YSlow: This firebug add-on from Firefox could be installed via the Internet from here: http://developer.yahoo.com/yslow/. Firebug add-on is a dependency for YSlow.
Page Speed: This is also a firebug add-on that can be downloaded and installed from: http://code.google.com/speed/page-speed/download.html.
For using GTmetrix and WebPagetest, we will need an active Internet connection. Make sure you have these.
How to do it...
Using the simple tool ab:
- If you are on a Windows environment, go to the apache/bin/ folder and if you are on Unix, fire up your terminal and issue the following command:
ab -c 10 -n 50 -g mage.tsv http://magento.local.com/
- In the previous command, the params denote the following:
- -c: This is the concurrency number of multiple requests to perform at a time. The default is one request at a time.
- -n: This requests the number of requests to perform for the benchmarking session. The default is to just perform a single request, which usually leads to non-representative benchmarking results.
- -g (gnuplot-file): This writes all measured values out as a gnuplot or TSV (tab separate values) file. This file can easily be imported into packages like Gnuplot, IDL, Mathematica, Igor, or even Excel. The labels are on the first line of the file.
- The preceding command generates some benchmarking report in the terminal and a file named mage.tsv in the current location, as we specified in the command.
- If we open the mage.tsv file in a spreadsheet editor such as Open Office or MS Excel, it should read as follows:
- You can tweak the ab params and view a full listing of params by typing ab -h in the terminal.
- Let's lay Siege to it! Siege is an HTTP regression testing and benchmarking utility. It was designed to let web developers measure the performance of their code under duress, to see how it will stand up to load on the Internet. Siege supports basic authentication, cookies, HTTP, and HTTPS protocols. It allows the user to hit a web server with a configurable number of concurrent simulated users. These users place the web server 'under Siege'.
- Let's create a text file with the URLs that would be tested under Siege. We can pass a single URL in the command line as well. We will use an external text file to use more URLs through a single command. Create a new text file in the terminal's current location. Let's assume that we are in the /Desktop/mage_benchmark/ directory. Create a file named mage_urls.txt here and put the following URLs in it:
- These URLs will vary with yours. Modify it as it fits. You can add more URLs if you want.
- Make sure that you are in the /Desktop/mage_benchmark/ directory in your terminal. Now issue the following command:
siege -c 50 -i -t 1M -d 3 -f mage_urls.txt
- This will take a fair amount of time. Be patient. After completion it should return a result something like the following:
Lifting the server siege.. done.
Transactions: 603 hits
Availability: 96.33 %
Elapsed time: 59.06 secs
Data transferred: 10.59 MB
Response time: 1.24 secs
Transaction rate: 10.21 trans/sec
Throughput: 0.18 MB/sec
Successful transactions: 603
Failed transactions: 23
Longest transaction: 29.46
Shortest transaction: 0.00
- Repeat the steps 1 and 3 to produce reports with some variations and save them wherever you want.
- The option details could be found by typing the following command in the terminal:
- Magento has a built-in profiler. You can enable it from the backend's System | Configuration | Advanced | Developer | Debug section.
- Now open the index.php file from your Magento root directory. Uncomment line numbers 65 and 71. The lines read as follows:
line 65: #Varien_Profiler::enable(); // delete #
line 71: #ini_set(<display_errors>, 1); // delete #
- Save this file and reload your Magento frontend in the browser. You should see the profiler data at the bottom of the page, similar to the following screenshot:
(Move the mouse over the image to enlarge.)
- We have already installed the YSlow firebug add-on. Open the Firefox browser and let's activate it by pressing the F12 button or clicking the firebug icon from the bottom-right corner of Firefox.
- Click on the YSlow link in firebug.
- Select the Rulesets. In my case I chose YSlow (V2).
- Click on the Run Test button.
- After a few seconds you will see a report page with the grade details. Here is mine:
- You can click on the links and see what it says.
- Fire up your Firefox browser.
- Activate the firebug panel by pressing F12.
- Click on the Page Speed link.
- Click on the Performance button and see the Page Speed Score and details. The output should be something like the following screenshot:
This is an online tool to benchmark a page with a combination of YSlow and Page Speed. Visit http://gtmetrix.com/ and DIY (Do It Yourself).
This is a similar tool as GTmetrix, which can be accessed from here: http://www.webpagetest.org/.
How it works...
ab is a tool for benchmarking your Apache Hypertext Transfer Protocol (HTTP) server. It is designed to give you an impression of how your current Apache installation performs. This especially shows you how many requests per second your Apache installation is capable of serving.
The analysis that Siege leaves you with can tell you a lot about the sustainability of your code and server under duress. Obviously, availability is the most critical factor. Anything less than 100 percent means there's a user who may not be able to access your site. So, in the above case, there's some issue to be looked at, given that availability was only 96.33 percent on a sustained 50 concurrent, one minute user Siege.
Concurrency is measured as the time of each transaction (defined as the number of server hits including any possible authentication challenges) divided by the elapsed time. It tells us the average number of simultaneous connections. High concurrency can be a leading indicator that the server is struggling. The longer it takes the server to complete a transaction while it's still opening sockets to deal with new traffic, the higher the concurrent traffic and the worse the server performance will be.
Yahoo!'s exceptional performance team has identified 34 rules that affect web page performance. YSlow's web page analysis is based on the 22 of these 34 rules that are testable. We used one of their predefined ruleset. You can modify and create your own as well. When analyzing a web page, YSlow deducts points for each infraction of a rule and then applies a grade to each rule. An overall grade and score for the web page is computed by summing up the values of the score for each rule weighted by the rule's importance. Note that the rules are weighted for an average page. For various reasons, there may be some rules that are less important for your particular page.
In YSlow 2.0, you can create your own custom rulesets in addition to the following three predefined rulesets:
- YSlow(V2): This ruleset contains the 22 rules
- Classic(V1): This ruleset contains the first 13 rules
- Small Site or Blog: This ruleset contains 14 rules that are applicable to small websites or blogs
Page Speed generates its results based on the state of the page at the time you run the tool. To ensure the most accurate results, you should wait until the page finishes loading before running Page Speed. Otherwise, Page Speed may not be able to fully analyze resources that haven't finished downloading.
Windows users can use Fiddler as an alternative to Siege. You can download it from http://www.fiddler2.com/fiddler2/, which is developed by Microsoft.
|Read more about this book|
(For more resources on Magento, see here.)
Optimizing Magento database and MySQL configuration
We already know the bottlenecks of our Magento store. Let us start with the Magento database and our MySQL server as it impacts drastically on the overall performance. In this recipe, we will optimize our store database as well as our MySQL server.
How to do it...
Optimizing the Magento database:
- This is a simple task. Launch your MySQL administration tool. phpMyAdmin is fine. Select your Magento database.
- You should see there are two pages for table structures. Click on the check all button from the bottom.
- Now keeping all tables selected, click on the Repair table from the drop-down besides the check all link.
- Do the same for the Optimize table from the drop-down.
- Repeat these steps for page 2 as well.
- At this stage, our database is good to go.
- Replicate your MySQL server.
- Split the write and read to master and slave database.
Optimizing MySQL server:
- Let us start from the system's operating system itself. To get the best use of multiple-CPU machines, you should use Linux (because 2.4 and later kernels have good SMP support). File systems such as ReiserFS and XFS do not have any size 2 GB limitation that is on ext2 though; ext2 could be patched for more than 2 GB size with the LFS patch. For running Magento, it's highly recommended to use a dedicated server or at least a VPS.
- If you have enough RAM, you could remove all swapped devices. Some operating systems use a swap device in some contexts even if you have free memory.
- Avoid external locking by using skip-external-locking in the [mysqld] section of my.cnf.
- The next important task is to set the key_buffer_size for MyISAM engine. Log in to your MySQL server with root access in the terminal.
mysql> SHOW VARIABLES LIKE '%key_buffer%';
- You should specify the current allocation for key_buffer_size. It is used by MyISAM tables to cache Index only, not data. If you have got a MyISAM system only, then the key_buffer_size should be 30 percent of memory. Remember that there is a limit of 4 GB per key buffer. MyISAM tables are used for temporary tables anyway. Let's set it as 512M. So, the following will be the MySQL command:
mysql> SET GLOBAL key_buffer_size = 536870912;
- We will set the rest of the configurations via the my.cnf file from the /etc/mysql/ directory.
- Open it and create the [mysqld] section as follows:
key_buffer = 512M
max_allowed_packet = 64M
thread_stack = 192K
thread_cache_size = 32
table_cache = 512
query_cache_type = 1
query_cache_size = 52428800
tmp_table_size = 128M
expire_logs_days = 10
max_binlog_size = 100M
sort_buffer_size = 4M
read_buffer_size = 4M
read_rnd_buffer_size = 2M
myisam_sort_buffer_size = 64M
wait_timeout = 300
max_connections = 400
- Save and restart your MySQL server.
How it works...
The most important tuning for any MySQL server are key_buffer, query_cache and table_cache. Make sure that you have specified the appropriate unit and value. Skip-external-locking is also an important tuning. It helps to deny any external locking.
You can view the current MySQL server status and variables by issuing the following commands:
You are encouraged to benchmark your server while applying these tunings. You should get about a 50 percent performance gain after these settings are applied.
Some wrong units in MySQL server tuning:
Wrong, table cache is measured in entries
Wrong again, key buffer should be specified in bytes
This one is set in percents
Optimizing Apache web server configuration
Now-a-days there are some other alternatives for Apache, but still the majority of us use Apache as a web server. The ultimate performance of any web application depends on some other factors as well such as CPU, network card, disk, OS, and RAM.
The most important hardware that affects any web server's performance is RAM. The more the better. In this recipe, we will optimize the web server, preferably a DS or at least a VPS.
How to do it...
- Update the operating system to its latest stable release. In most cases, OS suppliers have introduced significant performance improvements to their TCP stacks and thread libraries.
- Update your sendfile(2) system call support with the latest patch if your box is Linux and its kernel is lower than 2.4. If your box has got a kernel higher than 2.4 then it's ok. This ensures Apache 2 to deliver static contents faster with lower CPU utilization.
- Keep other background applications minimum. For example, in Unix, switch off NFS, any printing services, and even sendmail if it's not needed. Under Windows, use the system control panel to optimize the system for applications and system cache, and optimize the system for performance.
- ReiserFS and XFS are some good filesystems to use for better disk I/O.
- Web server should never ever have to swap, as swapping increases the latency of each request. You should control the MaxClients setting so that your server does not spawn so many children, that it starts swapping. Determine the size of your average Apache process, by looking at your process list via a tool such as top, and divide this into your total available memory, leaving some room for other processes.
- Set HostnameLookups to Off, if your Apache is lower than 1.3. If Apache is higher than 1.3, then HostnameLookups is defaulted to Off.
- It is highly discouraged to use SymLinksIfOwnerMatch, you can rather use it as Options FollowSymLinks + SymLinksIfOwnerMatch for specific directories. And for other locations use Options FollowSymLinks. This will prevent lstat(2) system calls. By the way, the results of lstat() are never cached.
- Never use DirectoryIndex as a wildcard, such as the following:
- Rather, use a complete one such as the following:
DirectoryIndex index.php index.html index.cgi index.pl index.shtml
- Enable mod deflate and mod header with the following command:
sudo a2enmod deflate
sudo a2enmod header
- Let us add some deflate settings in our .htaccess of our Magento store. Find and open the .htaccess from Magento root and modify it to show the deflate block as follows:
## enable apache served files compression
# Insert filter on all content
# Insert filter on selected content types only
AddOutputFilterByType DEFLATE text/html text/plain text/xml
# Netscape 4.x has some problems...
BrowserMatch ^Mozilla/4 gzip-only-text/html
# Netscape 4.06-4.08 have some more problems
BrowserMatch ^Mozilla/4\.0 no-gzip
# MSIE masquerades as Netscape, but it is fine
BrowserMatch \bMSIE !no-gzip !gzip-only-text/html
# Don't compress images
SetEnvIfNoCase Request_URI\.(?:gif|jpe?g|png)$ no-gzip dontvary
# Make sure proxies don't deliver the wrong content
Header append Vary User-Agent env=!dont-vary
- The next task is to set up mod expires. Issue the following command in your Linux terminal:
sudo a2enmod expires
sudo /etc/init.d/apache2 restart
- We must add expires settings in our .htaccess. Find the expires block and make it as follows:
## Add default Expires header
ExpiresDefault «access plus 1 year»
ExpiresByType image/gif «access plus 30 days»
ExpiresByType image/jpg «access plus 30 days»
ExpiresByType image/jpeg «access plus 30 days»
ExpiresByType image/png «access plus 30 days»
ExpiresByType image/x-icon «access plus 30 days»
ExpiresByType text/css «access plus 30 days»
</IfModule> Apache 1.1 comes with the Keep-Alive support on by
- Apache 1.1 and higher has come up with it's KeepAlive directive as On by default. Make sure that the KeepAlive is On. This is a trick where multiple HTTP requests can be funneled through a single TCP connection. You can modify it from the /etc/apache2/apache2.conf file.
- Set up your MPM as best. Experiment with it. Here is my configuration, this might and should be different depending on your system's resources:
- Benchmark your Magento again and see your changes in action!
How it works...
The overall performance of a LAMP application depends on the factors described previously and obviously your application itself. If your application cannot serve 100 concurrent hits then trying to optimize it is sort of a waste of time. If you are doing it on your own application other than Magento then make sure your application's code is able to serve at least 100 concurrent hits.
Server software, hardware also has a great impact on overall performance, so make it as high as possible. For overcoming common bottlenecks of a web application, we have some predefined rulesets that are described elaborately in Yahoo! and Google for YSlow and Page Speed, respectively. Follow them and you are good to go.
Last but not least, is to host your application in a server located in the same area as your targeted customer's location.
Tuning Magento configurations
Magento configuration is defaulted to run in a wide variety of servers, which might hinder the best performance. This recipe will help you tune the Magento configurations to let it perform fast and furious!
How to do it...
- Log in to your Magento backend as admin.
- Cleanup (uninstall and delete) all extensions/modules that you are not using.
- Enable all types of caching from the Magento backend | System | Cache Management page. Don't forget to flush/clear all cache storages.
- Compile your Magento from the System | Tools | Compilation | Run Compilation process. It compiles all Magento installation files and creates a single include path. It will speed up pages 25 to 50 percent, according to the official documentation.
- Go to the page System | Configuration | Advanced | Disable Modules Output and disable those which you are not using such as Mage_poll.
- Navigate to the page System | Configurations | Advanced | Developers and make it read as follows:
- Navigate to the Catalog | Attributes | Manage Attributes page. Set only those attribute frontend properties to Yes that you're actually going to use. Set all others to No. Don't use them in quick search, advanced search compare, and so on.
- Use store content's mark-up as W3 valid and avoid expressions in CSS like this:
background-color: expression( (new Date()).getHours()%2 ?
"#B8D4FF" : "#F08A00" );
How it works...
This article described Magento's performance optimization with some easy and incredible recipes dealing with YSlow, Page Speed, Siege, Apache bench, Apache configuration.
- Magento 1.4 Themes Design [Book]
- Magento 1.4 Development Cookbook [Book]
- Integrating Twitter with Magento [Article]
- Integrating Facebook with Magento [Article]
- How to Overcome the Pitfalls of Magento [Article]
- Getting Started with Magento Development [Article]