Troubleshooting Storage Contention

Exclusive offer: get 50% off this eBook here
Troubleshooting vSphere Storage

Troubleshooting vSphere Storage — Save 50%

Become a master at troubleshooting and solving common storage issues in your vSphere environment with this book and ebook

$23.99    $12.00
by Mike Preston | November 2013 | Enterprise Articles

This article written by Mike Preston, the author of the book Troubleshooting vSphere Storage, identifies storage contention and performance issues.

(For more resources related to this topic, see here.)

Now that we have learned about the various tools we can use to troubleshoot vSphere Storage and tackled the common issues that appear when we are trying to connect to our datastores, it's time for us to look at another type of issue with storage: contention. Storage contention is one of the most common causes of problems inside a virtual environment and is almost always the cause of slowness and performance issues.

One of the biggest benefits of virtualization is consolidation: the ability to take multiple workloads and run them on a smaller number of systems, clustered with shared resources, and with one management interface. That said, as soon as we begin to share these resources, contention is sure to occur. This article will help with some of the common issues we face pertaining to storage contention and performance.

Identifying storage contention and performance issues

One of the biggest causes of poor storage performance is quite often the result of high I/O latency values. Latency in its simplest definition is a measure of how long it takes for a single I/O request to occur from the standpoint of your virtualized applications. As we will find out later, vSphere further breaks the latency values down into more detailed and precise values based on individual components of the stack in order to aid us with troubleshooting.

But is storage latency always a bad thing? The answer to that is "it depends". Obviously, a high latency value is one of the least desirable metrics in terms of storage devices, but in terms of applications, it really depends on the type of workload we are running. Heavily utilized databases, for instance, are usually very sensitive when it comes to latency, often requiring very low latency values before exhibiting timeouts and degradation of performance.

There are however other applications, usually requiring throughput, which will not be as sensitive to latency and have a higher latency threshold. In all cases, we as vSphere administrators will always want to do our best to minimize storage latency and should be able to quickly identify issues related to latency.

As a vSphere administrator, we need to be able to monitor latency in our vSphere environment. This is where esxtop can be our number one tool. We will focus on three counters: DAVG/cmd, KAVG/cmd, and GAVG/cmd, all of which are explained in the following table:

When looking at the thresholds outlined in the preceding table, we have to understand that these are developed as more of a recommendation rather than a hard rule. Certainly, 25 ms of device latency isn't good, but it will affect our applications in different ways, sometimes bad, sometimes not at all.

The following sections will outline how we can view latency statistics as they pertain to disk adapters, disk devices, and virtual machines.

Disk adapter latency statistics

By activating the disk adapter display in esxtop, we are able to view our latency statistics as they relate to our HBAs and paths. This is helpful in terms of troubleshooting as it allows us to determine if the issue resides only on a single HBA or a single path to our storage array, as shown in the following screenshot:

Use the following steps to activate the disk adapter latency display:

  1. Start esxtop by executing the esxtop command.
  2. Press d to switch to the disk adapter display.
  3. Press f to select which columns you would like to display.
  4. Toggle the fields by pressing their corresponding letters. In order to view latency statistics effectively, we need to ensure that we have turned on Adapter Name (A), Path Name (B), and Overall Latency Stats (G) at the very least.

esxtop counters are also available for read and write latency specifically along with the overall latency statistics. This can be useful when troubleshooting storage latency as you may be experiencing quite a bit more write latency than read latency which can help you isolate the problems to different storage components.

Disk device latency statistics

The disk device display is crucial when troubleshooting storage contention and latency issues as it allows us to segregate any issues that may be occurring on a LUN by LUN basis.

Use the following steps to activate the disk device latency display:

  1. Start esxtop by executing the esxtop command.
  2. Press u to switch to the disk device display.
  3. Press f to select which columns you would like to display.
  4. Toggle the fields by pressing their corresponding letters. In order to view latency statistics effectively, we need to ensure that we have turned on Device Name (A) and Overall Latency Stats (I) at the very least.

By default, the Device column is not long enough to display the full ID of each device. For troubleshooting, we will need the complete device ID. We can enlarge this column by pressing L and entering the length as an integer that we want.

Virtual machine latency statistics

The latency statistics displayed inside the virtual machine display are not displayed using the same column headers as the previous two views. Instead, they are displayed as LAT/rd and LAT/wr. These counters are measured in milliseconds and represent the amount of time it takes to issue an I/O request from the virtual machine. This is a great view that can be used to determine a couple of things. One, is it just one virtual machine that is experiencing latency? And two, is the latency observed on mostly reads or writes?

Use the following steps to activate the virtual machine latency display:

  1. Start esxtop by executing the esxtop command.
  2. Press v to switch to the virtual machine disk display.
  3. Press f to select which columns you would like to display.
  4. Toggle the fields by pressing their corresponding letters. In order to view latency statistics effectively, we need to ensure that we have turned on VM Name (B), Read Latency Stats (G), and Write Latency Stats (H).

Summary

Storage contention and performance issues are one of the most common causes of slowness and outages within vSphere. Due to the number of software and hardware components involved in the vSphere storage stack, it's hard for us to pinpoint exactly where the root cause of a storage contention issue is occurring. Using some of the tools, examples, features, and common causes explained in this article, we should be able to isolate issues, making it easier for us to troubleshoot and resolve problems.

Resources for Article :


Further resources on this subject:


Troubleshooting vSphere Storage Become a master at troubleshooting and solving common storage issues in your vSphere environment with this book and ebook
Published: November 2013
eBook Price: $23.99
Book Price: $39.99
See more
Select your format and quantity:

About the Author :


Mike Preston

Mike Preston is an IT professional and an overall tech enthusiast living in Ontario, Canada. He has held all sorts of IT titles over the last 15 years including Network Technician, Systems Administrator, Programmer Analyst, Web Developer, and Systems Engineer in all sorts of different verticals, from sales to consulting. Currently, he is working as a Systems Analyst supporting the education market near his home in Belleville, Ontario.

Mike has always had an intense passion for sharing his skills, solutions, and work with various online communities, most recently focusing on the virtualization communities. He is an avid blogger at blog.mwpreston.net and participates in many discussions on Twitter (@mwpreston). It's his passion for sharing within the virtualization community which has led to Mike receiving the vExpert award for 2012 and 2013.

Mike has presented at VMworld, VMUGs, and various virtualization conferences on numerous times both as a customer and an overall evangelist and has published different whitepapers and articles for various tech websites. His commitment to giving back to the community has resulted in his most recent venture of becoming a Toronto VMUG co-leader. He is a VMware Certified Professional in Datacenter Virtualization on both Version 4 and 5 of vSphere and is currently pursuing his VCAP5-DCA, which he hopes to accomplish by 2014.

Books From Packt


vSphere High Performance Cookbook
vSphere High Performance Cookbook

VMware vSphere 5.1 Cookbook
VMware vSphere 5.1 Cookbook

Instant Citrix XenDesktop 5 Starter [Instant]
Instant Citrix XenDesktop 5 Starter [Instant]

VMware View 5 Desktop Virtualization Solutions
VMware View 5 Desktop Virtualization Solutions

VMware View Security Essentials
VMware View Security Essentials

Instant VMware vCloud Starter [Instant]
Instant VMware vCloud Starter [Instant]

IBM WebSphere Application Server v7.0 Security
IBM WebSphere Application Server v7.0 Security

Away3D 3.6 Essentials
Away3D 3.6 Essentials

No votes yet

Post new comment

CAPTCHA
This question is for testing whether you are a human visitor and to prevent automated spam submissions.
m
H
b
Q
8
Y
Enter the code without spaces and pay attention to upper/lower case.
Code Download and Errata
Packt Anytime, Anywhere
Register Books
Print Upgrades
eBook Downloads
Video Support
Contact Us
Awards Voting Nominations Previous Winners
Judges Open Source CMS Hall Of Fame CMS Most Promising Open Source Project Open Source E-Commerce Applications Open Source JavaScript Library Open Source Graphics Software
Resources
Open Source CMS Hall Of Fame CMS Most Promising Open Source Project Open Source E-Commerce Applications Open Source JavaScript Library Open Source Graphics Software