vSphere Design Best Practices

By Brian Bolander , Christopher Kusek
    What do you get with a Packt Subscription?

  • Instant access to this title and 7,500+ eBooks & Videos
  • Constantly updated with 100+ new titles each month
  • Breadth and depth in over 1,000+ technologies
  1. Virtual Data Center Design

About this book

vSphere allows you to transform your IT infrastructure into a private cloud, then bridge it to public clouds on-demand, delivering an IT infrastructure as an easily accessible service. vSphere delivers uncompromised control over all IT resources with the highest efficiency and choice in the industry.

The book begins with a definition of the core technologies in a virtual datacenter, vCenter, and ESXi. It then covers the architecture of specific virtual datacenter components. Readers will learn design principles related to storage and storage protocols. Moving on to networking, readers will learn to design flexible and reliable networks for their virtual datacenters. After this, Virtual Machine design considerations are reviewed in depth and readers are guided through inspecting existing VMs and design principles for correctly resourced and configured virtual machines.

Publication date:
May 2014


Chapter 1. Virtual Data Center Design

The datacenter has quite simply become one of the most critical components to organizations today. Some datacenters, either owned by large enterprises or leased by large service providers, can contain great amounts of bleeding-edge technologies, tens of thousands of servers, huge network pipes from multiple carriers, and enough contingency mechanisms to keep their systems running for months in case of a disaster. Other datacenters can consist of just a small handful of servers and off-the-shelf networking gear stuffed in a closet. Every organization is a bit different and has its own unique set of requirements in order to provide IT services to its users. As system administrators, engineers, and architects, we need to be able to take those requirements, sometimes even search for or define them, and design a solution to meet or exceed those requirements.

As datacenters have evolved, we've seen a paradigm shift towards virtualization. This is due to a number of factors including performance, availability, management, and recoverability. This book aims to help call attention to these factors and explains how to address them in your Virtual Data Center designs.

The Virtual Data Center has several key components such as compute, storage, networking, and management.

We will be covering the following topics in this chapter:

  • Core design principles for the Virtual Data Center

  • Best Practices for Virtual Data Center design

  • How best practices change over time

  • Virtual Data Center design scenarios

  • vCenter design including the Linux-based vCenter Server Appliance (vCSA)

  • vSphere clustering, HA, and DRS

  • A consideration of the other components of the vCloud Suite


Virtual Data Center design principles

VMware vSphere is a leading platform for virtualization with many components that make up the VMware vCloud Suite including vCenter Server, ESXi hypervisor, and Site Recovery Manager (SRM). Through these products, the vSphere platform, with proper design characteristics, enables an IT infrastructure to provide services, availability, flexibility, recoverability, and performance that its customers require.

Apart from knowledge of the aforementioned VMware products (or perhaps any third-party solutions being considered), there are a few key principles to get started with our designs. While there are numerous methodologies out there, this is a framework on which you can base your process consisting of three basic phases—Conceptual Design, Logical Design, and Physical Design , as shown in the following diagram:

It is important to remember that while the process is important, the focus of this book is about the design decisions. If you typically subscribe to a different framework or methodology, that is OK, as the design decisions discussed throughout the book will hold true regardless of your process and methodology in most cases.

First, create a Conceptual Design by gathering and analyzing business and application requirements, and then document the risks, constraints, and assumptions. Once all of the information is gathered and compiled, you should be able to formulate a high-level end goal, which is ultimately a vision of the solution. For example, you could have requirements to provide 99.99 percent uptime to an application, guarantee 2500 IOPS via shared storage, 25 GHz CPU, and 48 GB of RAM. You could be constrained by a budget of $100,000 among various risks and assumptions. So, conceptually, you'll have a high-level idea that you'll need a certain class of compute, storage, and network to support that type of application.

You certainly don't need to know the specifics at this point but you should have a vision and direction of the goal.

Next, building from the conceptual design, we'll begin formulating the Logical Design. To do this, we'll begin laying down the logical representations of the infrastructure components. Why just logical representations? Continuing with the above example, we don't know what servers we need or how many, we just know that we need servers. We don't know what storage array we need or what size, we just know that we need shared storage, and so on. We need to map each of the requirements to a logical solution within the complete design. This also begins to flush out dependencies between components and services. While it would be great if we could jump straight from the conceptual design right to the complete solution, we should be taking smaller steps towards our goal. This step is all about putting ideas down on paper and thinking through this Logical Design before we start procuring hardware and building up the datacenter by trial and error.

Finally, we transition our Logical Design into a Physical Design. While the previous steps allow us to be somewhat theoretical in our work, this phase absolutely requires us to do our homework and put specific parameters around our design. For example, we determine we need six servers with dual 8-core CPUs, 128 GB RAM, 15 TB of array-based fiber channel attached storage, 4 x 10 Gbe NICs, 2 x 8 Gb Fiber Channel HBAs, and so on. In most cases, you'll have this detail down to the manufacturer of each component as well. This physical design should consist of all the infrastructure components that will be required to support the requirements gathered in the Conceptual Design phase. For example, a complete Physical Design might include a vSphere Network Design, Storage Design, Compute Design, Virtual Machine (VM) Design, and Management Design. As you can imagine, the documentation for a physical design can get quite large and time consuming, but this is a critical step.

In summary, remember that the three design phases are additive and dependent upon one another. In order to have a successful Physical Design, both the Conceptual and Logical Designs must be solid. Document as much as is reasonable in your designs and even include reasoning behind choices. Many choices seem obvious while making them; however, when it is time to review and deploy a design months later, it may not be as obvious as to why certain choices were made.


Best practices

We hear the term Best practices everywhere but what does it really mean? Do we always adhere to best practices? What happens to best practices as technology evolves, or as your business evolves? These are great questions and ones that many people don't seem to ask. There are three scenarios when addressing a design.

You may ignore best practices, strictly follow best practices, or employ a combination of the two. The third scenario is the most common and should be your target. Why? The simplified answer is that there are certainly times when best practices aren't in fact best for your design. A great example is the general best practice for vSphere where the defaults for just about everything in the environment are acceptable. For example, take storage multipathing. VMware default is for fixed path and if best practice is default, you may consider not modifying this; experience shows that often you'll see performance gains switching to Round Robin (see VMware KB1011340 for details). While changing this default may not be a best practice, then again, "Best Practice" might not fit your needs in the real world. Choose what works for you and the constraints of your environment.

Best practices also seem to have a shelf life much longer than intended. A favorite example is that the best size for vSphere datastore is 500 GB. While this sizing best practice was certainly true at one time, vSphere has improved how it handles datastore sizing using features such as SCSI Reservation, which enables much higher VM per datastore density. Since vSphere 5.0, we've been able to create 64 TB datastores and the best practice is more about evaluating parameters such as number of VMs, number of VMDKs per VM, required IOPS, SLAs, fault domains, and restore capabilities. Many a times the backend disk capabilities will determine the ideal datastore size. Yet, many customers continue to use 500 GB as their standard datastore size. There are many more examples but the bottom line is that we should use best practices as a guideline and then do our homework to determine how to apply them to our designs.

Along with the best practices and design framework we've already discussed, it is also important to define and adhere to standards throughout your Virtual Data Center. This includes items such as naming conventions for ESXi hosts and VMs as well as IP address schemes, storage layout, network configurations, and security policies. Adhering to these standards will make understanding the design much easier as well as help as changes are introduced into the environment after deployment.


Designing Virtual Data Center

As previously mentioned VMware's Virtual Data Center (vDC) is composed of several key components: vCenter, ESXi hypervisor, CPU, storage, networking, and virtual machines. Many datacenters will have additional components but these form the foundation of typical vDC. Note that it is possible to operate, albeit in a limited capacity, without vCenter in the management layer. We find that it is really the core of the vDC and enables many features we seem to take for granted, such as vMotion, DRS (Distributed Resource Scheduling), and HA (High Availability).

Our vDCs will also consist of several constructs that we will leverage in our designs to provide organization and operability. These include the following:

  • Datacenters

  • Clusters

  • Resource pools

  • Folders

  • Tags

By using these constructs, we can bring consistency into complex designs. For example, by using folders and tags, we can organize VMs, hosts, and other objects so we can easily find them, report on them, or even perform operations such as backup based on folders or tags.


Tags are a new feature in vSphere 5.1 and above that work much like tagging in other applications. It allows us to put arbitrary tags on objects in our vDC for organizational purposes. Folders present a challenge when we have objects that could belong in multiple folders (which isn't possible). Many administrators use a combination of folders and tags to both organize their inventory as well manage it.

The following screenshot is a specific example of how folders can be used to organize our vDC inventory:

It is important to define a standard folder hierarchy as part of your design and include tags into that standard to help organize and manage your infrastructure.

Another facet to our designs will be how we might employ multiple vCenters and how we manage them. Many organizations utilize multiple vCenters because they want to manage Production, QA, and Test/Dev, separately. Others have vCenters distributed geographically with a vCenter managing each local infrastructure. In any case, with vSphere 5.5, multiple vCenter management has become much easier. We have what is called Linked Mode as an option. This allows us to provide a single pane of access to resources connecting to multiple vCenters—think shared user roles/groups and being able to access objects from any vCenters that are linked through a single console. However, with the vSphere web client included with vSphere 5.1 and above, we are able to add multiple vCenters to our view in the client. So, if your requirements are to have less roles and groups to manage, then Linked Mode may be the best route. If your requirements are to be able to manage multiple vCenters from a single interface, then the standard vSphere web client should meet your needs without adding the complexity of Linked Mode.

Next, we have Single sign-on (SSO). This is a new feature introduced in vSphere 5.1 and improved in vSphere 5.5. Previously, SSO had a difficult installation process and some difficult design decisions with respect to HA and multisite configurations. With SSO in vSphere 5.5, VMware has consolidated down to a single deployment model, so we're able to much more easily design HA and multisite configurations because they are integrated by default.

Finally, there is the subject of SSL certificates. Prior to vSphere 5.0, it was a best practice (and a difficult task) to replace the default self-signed certificates of the vSphere components (vCenter, ESX, and so on) with CA-signed certificates. That remains the case, but fortunately, VMware has developed and released the vCenter Certificate Automation tool, which greatly reduces the headaches caused when attempting to manually generate and replace all of those SSL certificates. Although not a steadfast requirement, it is certainly recommended to include CA-signed certificates in your designs.


VMware vCenter components

There are four primary components to vCenter 5.1 and 5.5:

  • Single sign-on: This provides identity management for administrators and applications that interact with the vSphere platform

  • vSphere web client: This is the service that provides the web-based GUI for administrators to interact with vCenter and the objects it manages

  • vCenter inventory service: This acts as a cache for vCenter managed objects when accessed via the web client to provide better performance and reduce lookups to the vCenter database

  • vCenter server: This is the core service of vCenter, which is required by all other components

There are also several optional components and support tools, which are as follows:

  • vSphere client: This is the "legacy" C#-based client that will no longer be available in the future versions of vSphere (although still required to interact with vSphere Update Manager)

  • vSphere Update Manager (VUM): This is a tool used to deploy upgrades and patches to ESXi hosts and VMware tools and virtual hardware version updates to VMs

  • vSphere ESXi Dump Collector: This is a tool used mostly to configure stateless ESXi hosts (that is, have no local storage) to dump VMkernel memory to a network location and then pull those logs back into vCenter

  • vSphere Syslog Collector: This is a tool similar to the Dump Collector that allows ESXi system logs to be redirected to a network location and be accessed via vCenter

  • vSphere Auto Deploy: This is a tool that can provision ESXi hosts over the network and load ESXi directly into memory allowing for stateless hosts and efficient provisioning

  • vSphere Authentication Proxy: This is a service that is typically used with Auto Deploy so that hosts can be joined to a directory service, such as MS Active Directory, without the need to store credentials in a configuration file

At the onset of your design, these components should be accounted for (if used) and integrated into the design. For small environments, it is generally acceptable to combine these services and roles on the vCenter while larger environments will generally want to separate these roles as much as possible to increase reliability by reducing the fault domain. This ensures that the core vCenter services have the resources they need to operate effectively.

Choosing a platform for your vCenter server

Now that we've reviewed the components of vCenter, we can start looking at making some design decisions. The first design decision we'll need to make relative to vCenter is whether you'll have a physical or virtual vCenter. VMware's own recommendation is to run vCenter as a VM. However, just as with best practices, we should examine and understand why before making a final decision. There are valid reasons for having a physical vCenter and it is acceptable to do. However, if we make the appropriate design decisions, a virtual vCenter can benefit from all the flexibility and manageability of being a VM without high amounts of risk.

One way to mitigate risks is to utilize a Management Cluster (sometimes called a management pod) with dedicated hosts for vCenter, DNS, Active Directory, monitoring systems, and other management infrastructure. This benefits us in a few ways such as being able to more easily locate these types of VMs if vCenter is down. Rather than connecting to multiple ESXi hosts via Secure Shell (SSH) or using the vSphere Client to connect to multiple hosts in order to locate a domain controller to be manually powered on, we're able to identify its exact location within our management pod. Using a management cluster is sometimes not an option due to the cost of a management pod, separate vCenter server for management, and the cost of managing a separate cluster. In smaller environments, it isn't such a big deal because we don't have many hosts to go searching around for VMs. But, in general, a dedicated cluster for management infrastructure is recommended.

A second way to mitigate risks for vCenter is to use a product called vCenter Heartbeat. VMware vCenter Heartbeat provides automated and manual failover and failback of your vCenter Server and ensures availability at the application and service layer. It can restart or restore individual services while monitoring and protecting the Microsoft SQL Server database associated with vCenter Server, even if it's installed on a separate server. For more information on vCenter Heartbeat, you can go to the product page at http://www.vmware.com/products/vcenter-server-heartbeat/.

In general, if choosing a virtual vCenter, do everything possible to ensure that the vCenter VM has a high priority with resources and HA restarts. An option would be to set CPU and memory resource shares to High for the vCenter VM. Also, do not disable DRS or use host affinity rules for the vCenter VM to try to reduce its movement within the vDC. This will negate many of the benefits of having a virtual vCenter.

Using the vCenter server appliance

Traditionally, this decision hasn't really required much thought on our part. The vCenter Server Appliance (vCSA or sometimes called vCVA or vCenter Virtual Appliance) prior to vSphere 5.5 was not supported for production use. It also had a limit of managing five hosts and 50 VMs. However, with vSphere 5.5, vCSA is fully supported in production and can manage 1000 hosts and 10000 VMs. See vSphere 5.5 Configuration Maximums at http://www.vmware.com/pdf/vsphere5/r55/vsphere-55-configuration-maximums.pdf.

So why would we want to use an appliance versus a full Windows VM? The appliance is Linux-based and comes with an embedded vPostgres, which will support up to 100 hosts and 3000 VMs. In order to scale higher, you need to use an external Oracle database server. Environments can save on Windows and MS SQL licensing by leveraging the vCSA. It is also a much simpler deployment. There are a few caveats, however, in that VMware Update Manager (VUM) still requires a separate Windows-based host with an MS SQL database as of vSphere 5.5. If you are running Horizon View, then you'll need a Windows server for the View Composer service and a similar situation exists if you are running vCloud Director. The general idea, though, is that the vCSA is easier to manage and is a purpose-built appliance. While vCSA may not be for everyone, but I would encourage you to at least consider it for your designs.

Sizing your vCenter server

If you're using the vCSA, this is less of an issue but there are still guidelines for disk space and memory as shown in the following table. Note that this only applies to vCSA 5.5 and above as the vCSA prior to 5.5 is only supported with a maximum of five hosts and 50 VMs.

VMware vCenter Server Appliance Hardware


Disk storage on the host machine

The vCenter Server Appliance requires at least 7 GB of disk space, and is limited to a maximum size of 80 GB. The vCenter Server Appliance can be deployed with thin-provisioned virtual disks that can grow to the maximum size of 80 GB. If the host machine does not have enough free disk space to accommodate the growth of the vCenter Server Appliance virtual disks, vCenter Server might cease operation, and you will not be able to manage your vSphere environment.

Memory in the VMware vCenter Server Appliance

  • Very small (≤ 10 hosts, ≤ 100 virtual machines): This has a size of at least 4 GB.

  • Small (10-100 hosts or 100-1000 virtual machines): This has a size of at least 8 GB.

  • Medium (100-400 hosts or 1000-4000 virtual machines): This has a size of at least 16 GB.

  • Large (≥ 400 hosts or 4000 virtual machines): This has a size of at least 24 GB.

Source: VMware KB 2052334

If you are choosing a Windows-based vCenter, you need to size your machine appropriately based on the size of your environment. This includes CPU, memory, disk space, and network speed/connectivity.


The number of clusters and VMs within a vCenter can absolutely affect performance. This is due to DRS calculations that must be made and more objects mean more calculations. Take this into account when estimating resource requirements for your vCenter to ensure performance and reliability.

It is also recommended to adjust the JVM (Java Virtual Machine) heap settings for the vCenter Management Web service (tcServer), Inventory Service, and Profile-driven Storage Service based on the size of deployment. Note that this only affects vCenter 5.1 and above.

The following VMware KB articles have up-to-date information for your specific vCenter version and should be referenced for vCenter sizing:

vCenter version

VMware KB article number







Choosing your vCenter database

The vCenter database is the location where all configuration information, along with some performance, task, and event data, is stored. As you might imagine, the vCenter DB can be a critical component to your virtual infrastructure. But it also can be disposable—although this is becoming a much less adopted strategy than it was a couple of years ago. In some environments that weren't utilizing features like the Virtual Distributed Switch (VDS) and didn't have a high reliance on the stored event data, we saw that vCenter could be quickly and easily reinstalled from scratch without much of an impact. In today's vDC's, however, we see more widespread use of VDS and peripheral VMware technologies that make the vCenter DB much more important and critical to keep backed up. For example, the VDS configuration is housed in the vCenter DB, so if we were to lose the vCenter DB, it becomes difficult and disruptive to migrate those hosts and their VMs to a new vCenter.

There are two other components that require a DB in vCenter and those are VUM and SSO. SSO is a new component introduced in vSphere 5.1 and then completely rewritten in vSphere 5.5. If you haven't deployed 5.1 or 5.5 yet, I'd strongly encourage you to go to 5.5 when you're ready to upgrade due to this re-architecture of SSO. In vSphere 5.1, the SSO setup requires manual DB configuration with SQL authentication and JDBC SSL, which caused many headaches and many support tickets with VMware support during the upgrade process.

We have the following choices for our vCenter DB:

  • MS SQL Express

  • MS SQL

  • IBM DB2

  • Oracle

The different versions of those DBs are supported depending on the version of vCenter you are deploying, so check the VMware Product Interoperability Matrixes located at http://partnerweb.vmware.com/comp_guide/sim/interop_matrix.php.

The MS SQL Express DB is used for small deployments—maximum of five hosts and 50 VMs—and is the easiest to set up and configure. Sometimes, this is an option for dev/test environments as well.

Next, we have probably the most popular option, that is, MS SQL. MS SQL can either be installed on the vCenter itself, but it is recommended to have a separate server dedicated to SQL. Again, sometimes in smaller deployments, it is OK to install the SQL DB on the vCenter server, but the general best practice is to have that dedicated database server for better protection, flexibility, and availability. With vCenter 5.5, we now have broader support of Windows Failover Clustering (formerly Microsoft Cluster Services or MSCS), which enhances our options for the availability of the vCenter DB.

Last, we have Oracle and IBM DB2. These two options are common for organizations that already have these databases in use for other applications. Each database type has its own benefits and features. Choose the database that will be the easiest to manage while providing the features that are required. In general, any of the database choices are completely capable of providing the features and performance of vCenter.

vSphere clustering – HA and DRS

Aside from vCenter itself, HA and DRS are two features that help unlock the true potential of virtualization. This combination of automatic failover and workload balancing result in more effective and efficient use of resources. The mechanics of HA and DRS are a full topic in and of itself, so we'll focus on how to design with HA and DRS in mind.

The following components will be covered in this section:

  • Host considerations

  • Networking design considerations

  • Storage design considerations

  • Cluster configuration considerations

  • Admission control

Host considerations

Providing high availability to your vSphere environment means making every reasonable effort to eliminate any single point of failure (SPoF). Within our hosts, we can make sure we're using redundant power supplies, ECC memory, multiple I/O adapters (such as NICs and HBAs), and using appropriate remote monitoring and alerting tools. We can also distribute our hosts across multiple racks or blade chassis to guard against the failure of a rack or chassis bringing down an entire cluster. So not only is component redundancy important, but it is also important to look at the physical location.

Some other items related to our hosts are using identical hardware. Although not always possible, such as hosts that have different lifecycles, we should make an effort to be consistent in our host hardware within a cluster. Aside from simplifying configuration and management, hardware consistency reduces resource fragmentation. For example, if we have hosts with Intel E5-2640 CPUs and E5-2690 CPUs, the cluster will become unbalanced and result in fragmentation. HA prepares for a worst-case scenario and plans for the largest host in a cluster to fail resulting in more resources being reserved for that HA event. This results in a lower efficiency in resource utilization.

The next consideration with HA is the number of hosts in your design. We typically look for an N+1 configuration for our cluster. N+1 represents the number of hosts we need to service our workload (N) plus one additional host in case of a failure. Some describe this as having a "hot spare", but in the case of vSphere, we're typically using all of the hosts to actively serve workload.


In some cases, it can be acceptable to actually reserve a host or hosts for failover. This is truly a hot spare scenario where those designated hosts sit idle and are used only in the case of an HA event. This is an acceptable practice, but it isn't the most efficient use of resources.

VMware vSphere 5.5 supports clusters of up to 32 hosts and it is important to understand how HA reserves resources based on cluster size. HA essentially reserves an entire hosts' worth of resources and then spreads that reservation across all hosts in the cluster. For example, if you have two hosts in a cluster, 50 percent of the total cluster resources are reserved for HA. To put it another way, if one host fails, the other host needs to have enough resource capacity to service the entire workload. If you have three hosts in a cluster, then 33 percent of the resources are reserved in the cluster for HA. You may have picked up on the pattern that HA reserves resources for the cluster where N is equal to the number of hosts in the cluster. As we increase the number of hosts in a cluster, we see a smaller percentage of resources reserved for HA that result in higher resource utilization. One could argue that larger clusters increase complexity but that is negated by the benefits of higher utilization. The important takeaway here is to remember that these reserved resources need to be available to the hosts. If they are being used up by VMs and an HA event occurs, it is possible that VMs will not be restarted automatically and you'll experience downtime.

Last, with respect to hosts, is versioning. It is absolutely recommended to have all hosts within a cluster at the same major and minor versions of ESXi, along with the same patch level. Mixed-host clusters are supported but because there are differences in HA, DRS, and other features, functional and performance issues can result. If different host versions are required, it is recommended to separate those hosts into different clusters, that is, a cluster for 5.0 hosts and a cluster for 5.5 hosts.

For a VMware HA percentage Admission Control Calculator see http://thinkingloudoncloud.com/calculator/vmware-ha-admission-control-calculator/.

Network considerations

Specific to HA, there are several design considerations regarding the network. Even though the dependency HA had on DNS has been removed in vSphere 5.0 and above, it is still a good practice to ensure all hosts and VMs can be resolved through DNS. Also, Spanning Tree Protocol (STP) on network switches can take a significant amount of time to start passing traffic on a port.


STP is a protocol that runs on network switches to prevent network loops.

For example, if a host is rebooted, STP takes approximately 50 seconds to recalculate the topology on its connected switch ports, and while that calculation is occurring the host does not have network connectivity. This can cause HA to report the host as dead and VMs will not start up on that host. This can also trigger an isolation response due to the datastore heartbeat. In either case, this can be prevented by enabling PortFast on those ports on the connected switches. Some switch vendors may call the feature something different, but the idea of PortFast is that STP will allow traffic to pass on those ports while it completes its calculations.

HA relies on the management network of the hosts. This brings with it several design considerations. Each host should be able to communicate with all other hosts' management interfaces within a cluster. Most of the time, this is not an issue because our network designs typically have all of our hosts' management VMkernel ports on the same layer 2 subnet, but there are scenarios where this is not the case. Therefore, it is important to make sure all hosts within a cluster can communicate properly to ensure proper HA functionality and prevent network partitions. Along these same lines, we should be designing our networks so that we have the fewest possible number of hops between hosts. Additional hops, or hardware segments, can cause higher latency and increase points of failure.

Because of the criticality of the management VMkernel port for proper HA functionality, it is imperative that we design in redundancy for this network. This means having more than one physical NIC (pNIC) in our hosts assigned to this network. If there is a failure in the management network, we still may get host isolation since HA won't have a means to test connectivity between vSphere servers.

When standardizing your network configuration, you should use consistent names for objects such as Port Groups, VLANs, and VMkernel ports. Inconsistent use of names across our hosts can result in VMs losing network connectivity after a vMotion or even preventing vMotion or an HA restart. This, of course, results in increased downtime.

Storage considerations

HA, as previously mentioned, also relies on datastore heartbeats to determine if an HA event has occurred. Prior to vSphere 5.0, we only had the network to determine whether a host had failed or become isolated. Now we can use datastores to aid in that determination enhancing our HA functionality. Datastore connectivity is inherently important—after all, our VMs can't run if they can't connect to their storage, right? So, in the case where we are utilizing a storage array, practices such as multipathing continue to be important and we need to make sure our hosts have multiple paths to our storage array whether that is iSCSI, NFS, FC, or FCoE.

vCenter will automatically choose two datastores to use for heartbeats. However, we do have the capability to override that choice. A scenario where this makes sense is if we have multiple storage arrays. It would be recommended to have a heartbeat datastore on two different arrays. Note that we can have up to five heartbeat datastores. Otherwise, the algorithm for choosing the heartbeat datastores is fairly robust and normally does not need to be tampered with. Another consideration is that if we are using all IP-based storage, we should try to have separate physical network infrastructure to connect to that storage to minimize the risk of the management network and storage network being disrupted at the same time. Last, wherever possible, all hosts in a cluster should have access to the same datastores to maximize the ability of isolated hosts to communicate via the datastore heartbeat.

Cluster considerations

Of the many configurations we have within a cluster for HA, Host Isolation Response seems to be often neglected. While we previously talked about how to make our management networks robust and resilient, it is still important to decide how we want our cluster to respond in the event of a failure. We have the following three options to choose from for the Host Isolation Response:

  • Leave Powered On: VMs will remain running on their host even though the host has become isolated

  • Power Off: VMs will be forcibly shutdown as if pulling the power plugs from a host

  • Shutdown: Using VMware tools, this option attempts to gracefully shutdown the VM and after a default timeout period of 300 seconds will then force the power off

In many cases, the management network may be disrupted but VMs are not affected. In a properly architected environment, host isolation is a rare occurrence, especially with a fiber channel SAN fabric as datastore heartbeats will be out-of-band of the network. If the design requires IP storage, which uses the same network infrastructure as the management network, then the recommended option is to shut down during isolation as it is likely that a disruption in the management network will also affect access to the VM's datastore(s). This ensures that another host can power on the affected VMs.

Another HA feature is VM and application monitoring. This feature uses VMware tools or an application agent to send heartbeats from the guest OS or application to the HA agent running on the ESXi host. If a consecutive number of heartbeats are lost, HA can be configured to restart the VM. The recommendation here is to utilize this feature and decide ahead of time how tolerant of heartbeat failures you wish to be.

Admission control

Admission Control (AC) is another great feature of HA. Because AC can prevent VMs from being powered on during a normal operation or restarted during an HA event, many administrators tend to turn AC off. HA uses AC to ensure that sufficient resources exist in a cluster and are reserved for VM recovery during an HA event. In other words, without AC enabled, you can overcommit your hosts to the point that if there was to be an HA event, you may find yourself with some VMs that are unable to power on. Let's take another look at a previous example where we have three hosts in a cluster. HA reserves 33 percent of that cluster for failover. Without AC enabled, we can still utilize that 33 percent reserve which puts us at risk. With AC, we are unable to utilize those reserved resources, which mitigates that risk of overprovisioning and ensures we'll have enough resources to power on our VMs during an HA event. When Admission Control is enabled, we can choose from the following three policies:

  • Host Failures Cluster Tolerates (default): This reserves resources based on the number of hosts for which we want to tolerate failures. For example, in a 10 host cluster, we could want to tolerate two host failures, so instead of the default one host failures the cluster tolerates (remember the formula), we would be allowing two host failures to tolerate in the cluster for failover.


    HA actually uses something called a "slot size" when determining resource reservations. Slot sizes are calculated on a worst-case basis; thus, if you have VMs with differing CPU and memory reservations, the slot size will be calculated from the VM with the highest reservations. This could greatly skew the slot size and result in more resources being reserved for failover than necessary.

  • Percentage of Cluster Resources Reserved: This reserves the specified percentage of CPU and memory resources for failover. If you have a cluster where VMs have significantly different CPU and memory reservations or hosts with different CPU and memory capacities, this policy is recommended. This setting must be revisited as the cluster grows or shrinks.

  • Specify a Failover Host: This designates a specific host or hosts as a hot spare.

The general recommendation is to use the percentage of cluster resources policy for Admission Control because it offers maximum flexibility. Design the percentage reserved to represent the number of host failures to be tolerated. For example, in a four host cluster, if you want to tolerate one host failure your percentage of failures tolerated is 25 percent, and if you wanted to tolerate two host failures it is 50 percent. If there will be hosts in the cluster of different sizes, be sure to use the largest host(s) as your reference for determining the percentage to reserve. Similarly, if the Specify a Failover Host policy is used, we must specify hosts that have CPU and memory equal to or larger than the largest nonfailover host in the cluster.

If the Host Failures Cluster Tolerates policy is to be used, then make every reasonable attempt to use similar VM resource reservations across the cluster. This is due to the slot size calculation mentioned earlier.



In this chapter, we learned about the design characteristics of the Virtual Data Center and the core components of the VMware vSphere platform. We also discussed best practices and design considerations specifically around vCenter, Clustering, and HA.

In the next chapter, we'll dive into the best practices of design of the ESXi hypervisor.

About the Authors

  • Brian Bolander

    Brian Bolander spent 13 years on active duty in the United States Air Force. A veteran of Operation Enduring Freedom, he was honorably discharged in 2005. He immediately returned to Afghanistan and worked on datacenter operations for the US Department of Defense (DoD) at various locations in southwest Asia. Invited to join a select team of internal consultants and troubleshooters responsible for operations in five countries, he was also the project manager for what was then the largest datacenter built in Afghanistan.

    After leaving Afghanistan in 2011, he managed IT operations in a DoD datacenter in the San Francisco Bay Area. His team was responsible for dozens of multimillion dollar programs running on VMware and supporting users across the globe.

    He scratched his adrenaline itch in 2011 when he went back "downrange", this time directing the premier engineering and installation team for the DoD in Afghanistan. It was his privilege to lead this talented group of engineers who were responsible for architecting virtual datacenter installations, IT project management, infrastructure upgrades, and technology deployments on VMware for the entire country from 2011 to 2013.

    Selected as a vExpert in 2014, he is currently supporting Operation Enduring Freedom as a senior virtualization and storage engineer.

    He loves his family, friends, and rescue mutts. He digs science, technology, and geek gear. He's an audiophile, a horologist, an avid collector, a woodworker, an amateur photographer, and a virtualization nerd.

    Browse publications by this author
  • Christopher Kusek

    Christopher Kusek lives in Portland, Oregon where he is Chief Technology Officer and Executive VP of Engineering at Xiologix. With over 20 years of experience in IT as a technology evangelist and industry leader, Christopher plays a key role in Xiologix's growth, leading its storage and engineering practice to evaluate and architect solutions that meet the client's tactical and strategic goals.

    He has over 20 years of experience in the industry with virtualization experience running back to the pre-1.0 days of VMware. He has shared his expertise with many far and wide through conferences, presentations, CXIParty, and sponsoring or presenting at community events and outings, whether it is focused on artificial intelligence, cloud, machine learning, networking, security, storage, or virtualization.

    Christopher has been a frequent contributor to VMware communities, vBrownBag, Twitter, and YouTube, and has been an active blogger for over a decade.
    Christopher is a proud VMware vExpert and a huge supporter of the program since its inception. He continues to help drive the growth of the virtualization community. He was named EMC Elect in 2013, 2014, 2015, and 2016, and continues with the renamed program of Dell EMC Elect in 2017. Christopher is a Cisco Champion 2016 and 2017, and has been a member of the VMware vExpert Program for nearly 10 years. Christopher is a vExpert specialist in the breakout designations of vExpert VSAN and vExpert NSX, and he's been a frequent contributor to VMware communities such as vBrownBag. Christopher shares his expertise online in the Thwack community as a SolarWinds ambassador, and he's a regular contributor and delegate to the Gestalt IT Tech Field Day series. You'll find Christopher directly through his Twitter handle, @cxi, and on his YouTube channel, CXI.

    Browse publications by this author
vSphere Design Best Practices
Unlock this book and the full library FREE for 7 days
Start now