System Center 2016 Virtual Machine Manager Cookbook - Third Edition

WAP is a free solution to manage resources that integrates with System Center and Windows Server to provide a customizable self-service portal for managing services such as websites, Virtual Machines, SQL or MySQL servers, and Service Bus; it also includes capabilities for automating and integrating additional custom services. For more info see http://www.microsoft.com/en-us/server-cloud/products/windows-azure-pack/.

Service Provider Foundation (SPF) is provided with System Center Orchestrator, a component of System Center since 2012 SP1. Service Provider Foundation exposes an extensible OData web service that interacts with Virtual Machine Manager (VMM). It's main interface for communication between WAP, SCOM, and VMM.

Service Reporting, an optional component of System Center 2012 R2, enables IT (particularly hosting providers) to create detailed views, for each customer (tenant), of the virtual machine's consumption of the resources (CPU, memory, storage, and networking). For more info see http://technet.microsoft.com/en-us/library/dn251058.aspx.

Although the domain controller is not part of the System Center family and it is not a VMM component, it plays an important role in the deployment of a private cloud as VMM requires it to be installed on a domain environment.

WSUS plays an important role with reference to the private cloud as it is used to update the Hyper-V hosts, library servers, or any other role for compliance and remediation.

The App Controller provides a self-service experience through a web portal that can help you easily configure, deploy, and manage VMs and services across private, third-party hosters (that support Microsoft Hyper-V) and public clouds (Azure). For example, moving a VM from a private cloud to Azure, creating checkpoints, granting access, scaling out deployed services, and so on.

The App Controller has been used as a replacement of the VMM self-service portal since SC 2012 SP1. It was deprecated in the SC 2012 R2 time and finally removed in SC 2016. As noted above, you should plan Azure Pack deployment instead of current App Controller instance.

Azure Stack is a hybrid-cloud platform, bringing core public Azure services to your datacenter. These services are mostly dedicated to Azure PaaS and IaaS and help you out with building unified ecosystems between private and public clouds. Azure Stack is delivered as an integrated system, with software installed on the hardware built by partners like HPE and Cisco. Azure's familiar pay-as-you-go model is mainly being used in Azure Stack and you can stretch the same subscriptions out for both Azure and Azure Stack clouds. If you have unstable or restricted connection to Azure, you may choose to use Azure Stack in disconnected mode with a capacity model pricing package - a fixed fee annual subscription based on the number of physical cores. It's important to note that you can manage WAP VMs from Azure Stack using a special connector, though it's under review and not recommended for production use: https://aka.ms/wapconnectorazurestackdlc.

The following table will guide you through choosing which System Center component is necessary as per your deployment:

In an enterprise deployment of VMM, it is recommended that you have a SQL Server cluster to support the HA VMM, preferably on a cluster separated from the VMM cluster. VMM 2016 supports SQL Server Always On Availability Groups. The following link will show you a good example of how to set it up: See the Configure SQL Server with AlwaysOn AGs recipe in Chapter 3, Installing VMM 2016.

As it is the best practice in an enterprise deployment, a highly available file server for hosting the VMM library shares is highly recommended as VMM does not provide a method for replicating files in the VMM library, and they need to be replicated outside of VMM.

As a suggestion, you can use the Microsoft Robocopy tool to replicate the VMM library files if you have distributed the library type.

VMM provides support for both Block level storage (Fibre Channel, iSCSI, and Serial Attached SCSI (SAS) connections) and File storage (on SMB 3.0 network shares, residing on a Windows file server or on a NAS device).

By using storage providers, VMM enables discovery, provisioning, classification, allocation, and decommissioning.

Storage classifications enable you to assign user-defined storage classifications to discovered storage pools for Quality of Service (QoS) or chargeback purposes.

In order to use this feature, you will need the SMI-S provider.

VMM 2016 can discover and communicate with SAN arrays through the Storage Management Initiative (SMI-S provider) and Storage Management Provider (SMP) provider.

If your storage is SMI-S compatible, you must install the storage provider on a separately available server (do not install on the VMM management server) and then add the provider to VMM management. Some devices come with built-in SMI-S provider and no extra are tasks required in that case. If your storage is SMP-compatible, it does not require a provider installation either.

CIM-XML is used by VMM to communicate with the underlying SMI-S providers since VMM never communicates with the SAN arrays themselves.

By using the storage provider to integrate with the storage, VMM can create LUNs (both GPT and MBR) and assign storage to hosts or clusters.

VMM 2016 also supports the SAN snapshot and clone feature, allowing you to duplicate a LUN through a SAN Copy-capable template to provide for new VMs, if you are hosting those in a Hyper-V platform. You will need to provision outside of VMM for any other VMs hosted with VMware hosts, for example.

This capability enables VMM 2016 to identify the hardware, install the operational system (OS), enable the Hyper-V or file server role, and add the machine to a target-host group with streamlined operations in an automated process.

PXE capability is required and is an integral component of the server pool. The target server will need to have a baseboard management controller (BMC) supporting one of the following management protocols:

Enterprise and hosting companies will benefit from the ability to provide new Hyper-V servers without having to install the operational system manually on each machine. By using BMC and integrating with Windows Deployment Services (WDS), VMM deploys the OS to designated hosts through the boot from the VHD(X) feature. The right BMC configuration presence is also a requirement for one of the most interesting features, called OS Rolling Upgrade, which will be discussed in detail later.

To ensure that users can perform only assigned actions on selected resources, create tenants, self-service users, delegated administrators, and read-only administrators in VMM using the VMM console, you will need to create Run As accounts to provide necessary credentials for performing operations in VMM ( example, for adding hosts).

Run As accounts are very useful additions to enterprise environments. These accounts are used to store credentials that allow you to delegate tasks to other administrators and self-service userswithout exposing sensitive credentials.

There are several different categories of Run As accounts:

Only administrators or delegated administrators can create and manage Run As accounts.

When designing the VMM implementation, you need to plan which ports you are going to use for communication and file transfers between VMM components. Based on the chosen ports, you will also need to configure your host and external firewalls. See the Configuring ports and protocols on the host firewall for each SCVMM component recipe in Chapter 3, Installing VMM 2016.

The recommendation is to create a big CSV volume. CSV spreads across multiple disk spindles and it will give great storage performance for VMs, as opposed to creating volumes based on the VHD purpose (for example, OS, data, and logs).

VMM 2016 supports management up to 1000 physical hosts and 25000 VMs. Therefore, the best practice is to have a separate management cluster with running VMM components to manage the production, test, and development clusters.

In addition to this, although you can virtualize the domain controllers with Windows 2016, it is not the best practice to have all the domain controllers running on the management clusters, as the cluster and System Center components highly depend on the domain controllers. If it's possible, place one or more DCs on the physical hosts or VMs in the location or fault domains different from the management cluster.

The following figure shows a two-node hyper-converged management cluster, with System Center 2016 components installed in separate VMs to manage the production cluster. All hosts are running on Windows Server 2016 with enabled Storage Spaces Direct to provide hyper-converged solutions which help to maximize the server's efficiency and reduce overall costs:

In a small environment, you can have all the VMM components located on the same server. A small business may or may not have high availability in place, as VMM 2016 is now a critical component for your private cloud deployment.

Start by selecting the VMM server's location, which could be a physical server or a virtual machine.

You can install SQL Server on the VMM server as well, but as VMM 2016 does not support SQL Express editions, you will need to install SQL Server first and then proceed with the VMM installation.

If you are managing more than 10 hosts in the production environment, my recommendation would be to have SQL Server running on a separate machine.

It is important to understand that when deploying VMM in production environments (real-world scenarios), the business will require a reliable system that it can trust.

The following figure illustrates a real-world deployment where all VMM 2016 components are installed on the same VM and SQL is running on a separate VM.

I would recommend up to 50 hosts in a lab environment with SQL Server and all VMM components installed on a single VM. It will work well, but I would not recommend this installation in a production environment.

In a medium-scale or large-scale environment, the best practice is to split the roles across multiple servers or virtual machines. By splitting the components, you can scale out and introduce high availability to the System Center environment.

In the following design, you can see each component and what role it performs in the System Center Virtual Machine Manager environment:

When designing an enterprise private cloud infrastructure, you should take into consideration some key factors such as business requirements, company policies, applications, services, workloads, current hardware, network infrastructure, storage, security, and users.

Following is a sample of a real-world infrastructure that can support up to 3000 VMs and 64 server nodes running Windows 2016 Hyper-V.

The number of VMs you can run on an implementation like this will depend on some key factors. Do not take the following configuration as a mirror for your deployment, but as a starting point. My recommendation is to start understanding the environment, then run a capacity planner such as a MAP toolkit. It will help you gather information that you can use to design your private cloud.

I am assuming a ratio of 50 VMs per node cluster with 3 GB of RAM, configured to use Dynamic Memory (DM):

System Center 2012 SP1 VMM introduced multi-tenancy. This is one of the most important features for hosting companies as they only need to install a single copy of System Center VMM, and then centralize their customer management, each one running in a controlled environment in their own domain. Hosters always want to maximize their compute capacity and VLAN segment hardware so you can't maximize its capacity. Network virtualization moves the isolation up to the software stack, enabling the hoster to maximize all capacity and isolate customers via software-defined networking VMM 2012 R2 takes advantage of Windows Server 2012 R2 features, VMM 2012 R2 delivers Site-to-Site NVGRE gateway for Hyper-V network virtualization. This capability enables you to use network virtualization to support multiple Site-to-Site tunnels and direct access through a NAT Firewall. The networking virtualization (NV) uses NVGRE protocol, allowing network load balancers to act as NV gateways. Plus, switch extensions can make use of NV policies to interpret the IP information in packets being sent and communication between, for example, Cisco switches and VMM 2012 R2.

VMM 2016 and Windows Server 2016 continue to improve Hyper-V Network-Virtualization (HNV) and helps you move to an efficient SDDC solution. VMM 2016 introduces flexible encapsulation which supports both NVGRE (HNVv1) and new VXLAN (HNVv2) to create overlay networks in which original packets from VMs with its MACs, IPs and other data (Customer Address network) are placed inside an IP packet on the underlying physical network (Provider Address network) for further transportation. VXLAN is the default in VMM 2016 and works in MAC distribution mode. It uses a new Network Controller (NC) as a central management point that communicates with Hyper-V hosts and pushes network policies down to NC host agents running on each host. In short, NC is responsible for the address mapping, and the host agents maintain the mapping database. NC also integrates with Software-Load Balancer (L3 and L4), network layer datacenter firewall and RAS gateways which are also included in Windows Server 2016. Consequently, NC is the heart of SDN in VMM 2016 and should always be considered in a cluster configuration.

There is also a new way of deploying converged networking that has been introduced in Windows Server 2016 and VMM 2016 to ease and improve SDN deployment. Switch-Embedded Teaming (SET) allows you to group up to eight identical adapters into one or more software-based virtual adapters. Prior to VMM 2016 you needed to have two different sets of adapters: one to use with traditional teaming and the one to use with RDMA because of its incompatibility with teaming and virtual switch. SET eliminates this limitation and supports RDMA convergence as well as QoS, RSS, VMQ, and both versions of HNV noted earlier. Furthermore, creating a general Hyper-V virtual switch with RDMA NICs would be also supported:

When we discussed possible architectures for management clusters, I referred to a new feature in Windows Server 2016 and VMM 2016, Storage Spaces Direct (S2D). S2D uses industry-standard servers with local storage which could be direct-attached enclosures or internal disks. S2D provides similar shared storage pools across cluster nodes by leveraging Cluster Shared Volume, Storage Spaces, Failover Clustering and SMB3 protocol for file access (SOFS). Hyper-converged and converged solutions can now be based on software-defined storage running on Windows Server 2016. So, you have a choice: to buy external enterprise SAN or to use S2D. If your goal is a software-defined datacenter, the answer to all questions is very clear - S2D and SDN implementation. The main competitor to S2D is a well-known VMware Virtual SAN (vSAN) that was first released in vSphere 5.5 and is still present in the newest vSphere 6.6. S2D, just like a vSAN, has special licensing requirements.

Furthermore, improved Storage QoS in VMM 2016 provides a way to centrally monitor and manage storage performance for virtual machines residing on S2D or another device. Storage QoS was first introduced in 2012 R2 version. You could set maximum and minimum IOPS thresholds for virtual hard disks (excluding shared virtual hard disks). It worked well on standalone Hyper-V hosts, but if you have a cluster with a lot of virtual machines or even tenants, it could be complicated to achieve the right QoS for all cluster resources. The feature automatically improves storage resource fairness between multiple virtual machines using the same file server cluster. In other words, QoS for storage will be distributed between a group of virtual machines and virtual hard disks:

Another feature available only in Windows Server 2016 Datacenter edition is Storage Replica (SR). Previously, we needed to find third-party solutions for SAN-to-SAN replication. And building stretched clusters required a huge amount of money. Windows Server 2016 and VMM 2016 can help to significantly reduce costs and enhance unification in such scenarios. SR is the main component of multi-site clusters or disaster recovery solutions supporting both asynchronous and synchronous replication between any storage devices, including Storage Spaces Direct. Also, you are not required to have identical devices on both sides. However, at the time of writing, only synchronous replication is supported in VMM fabric, and deployment is limited to PowerShell.

Following are the hardware requirements to consider when specifying your VMM environment. Although for SMB, POC or demo scenarios you can have SQL installed on the VMM management server, the recommendation is to have SQL Server installed on another server. And you also won't run SQL and Library Servers on the VMM server if you want to manage more than 150 hosts.

Following are the hardware requirements for VMM management server:

Following are the hardware requirements for VMM database server:

Following are the hardware requirements for VMM library server.

The minimum and recommended requirements for a VMM library server will be determined by the quantity and size of the files that will be stored:

Following are the hardware requirements for the VMM console:

Following are the hardware requirements for the Windows Azure Pack.

Before the WAP installation, you need to consider which type of deployment and components you really need. In an express deployment, all core components are installed on the same machine. This type of deployment is recommended for demo and POC scenarios. Distributed deployment is when WAP portals and databases are running on dedicated virtual machines (up to 8, except for VMs for optional resource providers like VM Clouds or Web Sites):

Following are the requirements for VMM management server for SC 2016:

The following table shows the requirements for the VMM console:

Following are the requirements for core WAP components:

You can check if the WinRM can communicate with OS WMI providers by running the following command:

winrm enum wmi/root/cimv2/Win32_ComputerSystem -r :http://servername:5985 [-u:YOURDOMAIN\AdminUser]

By running the following command, you can check if the WinRM can communicate with Hyper-V WMI providers:

winrm enum wmi/root/virtualization/v2/msvm_computersystem   -r:http://servername:5985 [-u:YOURDOMAIN\AdminUser]

Also, to check if the WinRM can communicate with the VMM agent WMI provider, run the following command:

winrm invoke GetVersion wmi/root/scvmm/AgentManagement -r:servername [-u:YOURDOMAIN\AdminUser] @{}

Following are the troubleshooting tools available for use:

              wmimgmt.msc

BITS transfers files between machines, providing information about the operation's progress. The transfer can be asynchronous.

In VMM, BITS is used for encrypted data transfer between managed computers. Encryption is done by using a self-signed certificate generated when the Hyper-V host is added to VMM.

You can use BITSadmin to verify that BITS is working properly outside of VMM.

BITSadmin is available in Windows Server. Some useful examples of BITSadmin are described at: http://msdn.microsoft.com/en-us/library/aa362812(VS.85).aspx.

The following tools are used to collect data surrounding VMM issues:

logman create trace VMMDebug -v mmddhhmm -o
$env:SystemDrive\VMMlogs\DebugTrace_$env:computername.ETL 
-cnf 01:00:00 -p Microsoft- VirtualMachineManager-Debug                           
-nb 10 250 -bs16 -max 512

logman start VMMDebug 

              logman stop VMMDebug 

              logman delete VMMDebug

After gathering the trace, you can use netsh trace convertcommand on the traces. This command converts the ETL binary trace logs into TXT files that can be viewed in any text editor. To convert the ETL file:

      nets  trace convert C:\VMMlogs\yourfilename.etl

You will receive a response like the following:

       Input file: C:\VMMlogs\VMMLog_VMM01_09242130.etl
       Dump file: C:\VMMlogs\VMMLog_VMM01_09242130.txt
       Dump format: TXT
       Report file: -
       Generating dump ... done

      notepad c:\VMMlogs\yourfilename.txt