VMware NSX Cookbook

By Bayu Wibowo , Tony Sangha
    What do you get with a Packt Subscription?

  • Instant access to this title and 7,500+ eBooks & Videos
  • Constantly updated with 100+ new titles each month
  • Breadth and depth in over 1,000+ technologies
  1. Free Chapter
    Getting Started with VMware NSX for vSphere
About this book
This book begins with a brief introduction to VMware's NSX for vSphere Network Virtualization solutions and how to deploy and configure NSX components and features such as Logical Switching, Logical Routing, layer 2 bridging and the Edge Services Gateway. Moving on to security, the book shows you how to enable micro-segmentation through NSX Distributed Firewall and Identity Firewall and how to do service insertion via network and guest introspection. After covering all the feature configurations for single-site deployment, the focus then shifts to multi-site setups using Cross-vCenter NSX. Next, the book covers management, backing up and restoring, upgrading, and monitoring using built-in NSX features such as Flow Monitoring, Traceflow, Application Rule Manager, and Endpoint Monitoring. Towards the end, you will explore how to leverage VMware NSX REST API using various tools from Python to VMware vRealize Orchestrator.
Publication date:
March 2018


Chapter 1. Getting Started with VMware NSX for vSphere

In this chapter, we will explore how to install and configure NSX for vSphere. We will be covering the following recipes:

  • Choosing the right VMware NSX for vSphere edition
  • Selecting ESXi hosts and network adapters
  • Downloading NSX for vSphere
  • Deploying the NSX Manager virtual appliance
  • Replacing the NSX Manager certificate
  • Registering vCenter server with NSX Manager
  • Applying the NSX licenses
  • Deploying the NSX Controller Cluster
  • Preparing a vSphere cluster for NSX
  • Validating NSX VIB installation
  • Checking NSX component communication


This book aims to be useful for both new and seasoned VMware NSX for vSphere administrators. It is intended to be used by those that have never deployed NSX and by those that have it deployed already but are looking to leverage newer or advanced functionality. Intermediate networking and virtualization knowledge is assumed and is essential to understanding deployment of NSX into your environment.

Before we begin serving the main recipes of our cookbook, we will first provide an overview of what VMware NSX for vSphere is and what functionality it provides over traditional networking models.

VMware NSX for vSphere is a core component of the VMware Software-Defined Data Center (SDDC); it is the component that enables network virtualization. Network virtualization provides a layer of abstraction over the physical network using a VXLAN network overlay. With NSX, network operations are now independent of the physical hardware, and functions such as logical firewalls, load balancers, logical routers, logical switches, and virtual private networks can be provisioned, modified, or torn down as part of an automated workflow.


Choosing the right VMware NSX for vSphere edition

VMware NSX has four licensing editions: standard, advanced, enterprise, and remote office/branch offices (ROBO). Each licensing tier provides distinctive functionality, available per CPU socket on a perpetual basis at the vSphere cluster level.

The standard and advanced editions are also available as per 100 users in a pack basis to align with virtual desktop deployments (vSphere for desktop). The enterprise edition is also available on per-VM term basis. You can upgrade from standard to advanced/enterprise and from advanced to enterprise.


Prior to NSX 6.2.2, VMware NSX for vSphere did not have multiple licensing tiers. If you purchased NSX prior to May 3, 2016, you are entitled to VMware NSX Enterprise edition as long as you have active support and subscription contracts. You can upgrade your VMware NSX license key from the My VMware portal (http://my.vmware.com).

Getting ready

Like vSphere licensing, VMware NSX is licensed per CPU socket. If you have a separate Management vSphere Cluster that is used for Infrastructure VMs and are not planning to protect it with the NSX Distributed Firewall or place NSX Edge Service Gateways onto it, you are not required to license the CPUs on that Management vSphere Cluster. The Compute vSphere cluster and Edge vSphere cluster need to be licensed.


VMware NSX is licensed at the vSphere Cluster level. If you need to exclude a specific ESXi host from NSX, you will need to remove the ESXi host from the cluster. For vSphere environments with VMware vCenter Site Recovery Manager, you will normally have active sites (Protected site) and passive/disaster recovery sites (Recovery site). Only the active ESXi hosts on the protected site requires a VMware NSX license. For more about licensing NSX for vSphere see VMware KB 2078615 (https://kb.vmware.com/kb/2078615).

How to do it...

From your vSphere inventory you will need to do the following:

  1. Determine how many CPU sockets you need
  2. Determine the NSX features required
  3. If you are planning to integrate third-party partner solutions with NSX (http://www.vmware.com/products/nsx/technology-partners.html), check whether a specific NSX feature is required


Some security services partner solutions require NSX distributed firewalling features and physical-to-virtual data center services requires integration with a Hardware VTEP (HW VTEP). 

  1. Choose the NSX edition based on the features required


I would like to use VMware vShield Endpoint for anti-virus/anti-malware capability only. Which NSX edition should I use? VMware vShield endpoint is included as a vSphere feature in the vSphere Essential Plus Edition or later, so you do not need to purchase VMware a NSX license. VMware NSX for vShield endpoint will appear on the vSphere download site if you have vSphere Essential Plus Edition or later. For more information, see VMware KB 2110078 (https://kb.vmware.com/kb/2110078).

There's more...

The following sub-sections will detail the different tiers of NSX licensing and the features available in each. From there, how to evaluate and purchase VMware NSX will also be detailed.

VMware NSX editions

The four tiers of licenses are as follows:

  • Standard edition
  • Advanced edition
  • Enterprise edition
  • ROBO

The features available in each edition are as follows:

Product feature





Distributed Switching

Distributed Routing

NSX Edge Firewall

Network Address Translation (NAT)

SW L2 Bridging to physical environment

Dynamic routing with ECMP (Active-Active)


Integration with vRealize and OpenStack

Automation of security policies with vRealize

NSX Edge Load Balancing

Distributed Firewalling

Integration with Active Directory

Service Insertion (third-party integration)

Cross vCenter NSX

Multisite NSX optimizations

VPN (IPSec and SSL)

Remote gateway

Integration with HW VTEPs


Distributed switching for the ROBO licensing tier is only available on VLAN-backed networks. Distributed load balancing is available in Enterprise edition as a tech preview.

Evaluating VMware NSX 

There are two ways to evaluate VMware products:

  • Deploy NSX in your environment and use an evaluation license for a limited time


VMware NSX license is not available publicly. Contact your VMware sales representative to get an NSX evaluation license.

Support and Subscription (SnS)

There are support and subscription plan options that you can purchase in addition to the product:

  • Basic support: 12 hours a day technical support during business hours
  • Production support: 24 hours (Severity 1), seven days a week support

The production support plan is recommended for production and critical environments. If you need higher-level support above production grade, additional support options such as Business Critical Support (BCS) or Mission Critical Support (MCS) can be purchased on top of production support. For more information on VMware support offerings, see https://www.vmware.com/support/services.html.

VMware vRealize Log Insight for NSX

VMware vRealize Log Insight is a log management engine that collects logs from a number of different sources and provides rich dashboards and search functionality.

Log Insight is available for NSX at no additional charge, you are entitled to one Log Insight CPU per NSX CPU license. The support and subscription is included with the NSX purchase. It is a fully functioning version of Log Insight but limited to vSphere and NSX data sources and content packs only. If you need more data sources and content packs, additional Log Insight licenses are required.

VMware NSX Monitoring Tools

There are several tools for monitoring VMware NSX. Some of these tools are built directly into the NSX platform, and others are separate feature-rich VMware products. These tools are as follows:

  • VMware NSX built-in tools
  • vRealize Network Insight

See also

For more information about the VMware NSX Neutron plugin license editions for VMware integrated OpenStack, see VMware KB 2145269 (https://kb.vmware.com/kb/2145269).


Selecting ESXi hosts and network adapters

Similar to the requirements of a VMware vSphere solution, choosing the correct hardware is still an important part of any NSX deployment; therefore, you need to follow the same process that you did for vSphere to ensure the hardware you are deploying is on the VMware Compatibility Guide (http://www.vmware.com/resources/compatibility/search.php).

The compatibility guide does not only list the supported servers, but you need to also check if your network interface card (I/O devices) is supported and features such as VXLAN Offload and Receive Side Scaling are also supported.

VXLAN Offload

VXLAN Offload is akin to TCP segmentation offload (TSO), but compared to TSO, which is designed for TCP packet headers, VXLAN encapsulates the original (source) packet from a virtual machine into a user datagram protocol (UDP) packet with its own unique header, known as the VXLAN header. Placing this additional header onto a packet invalidates traditional offloading mechanisms in-place and therefore increases load on the CPU as additional CPU cycles are needed to encapsulate and decapsulate every VXLAN packet. VXLAN is covered in greater detail in Chapter 2,Configuring VMware NSX Logical Switch Networks.

Receive Side Scaling

Receive Side Scaling (RSS) is a technique the Network Interface Card (NIC) employs to ensure that data processing for a particular connection is balanced across multiple CPU cores. Without RSS, all connections would be handled by a single CPU core, which can adversely affect network performance.


When using VMware Compatibility Guide, it is important to check the Network Interface Card supports VXLAN Offload and RSS; this will ensure that ESXi is able to leverage native hardware offloading for increased performance. This is only required if you are using VXLAN in your NSX deployment.


Downloading NSX for vSphere

In this recipe, we will download the installation media for NSX for vSphere. The installation media comes in the form of an open virtual application (OVA) that is distributed through the VMware downloads site (https://my.vmware.com/web/vmware/downloads).

Getting ready

To download NSX for vSphere, the following prerequisites must be satisfied:

  • Valid VMware software entitlements that enable you to download the installation media
  • Access to the VMware downloads website
  • Access to VMware software manager. Download and install VMware software manager first (https://www.vmware.com/go/download-software-manager-en)
  • VMware product interoperability matrix has been consulted so you know which version is compatible with your environment


Where can I download VMware NSX for vShield endpoint?vSphere Essential Plus and later editions come with vShield endpoint. VMware NSX will appear on the vSphere download site similar to vCNS (vCloud Networking and Security).

How to do it...

The following sections will explain how to check that your infrastructure supports the version of NSX you are implementing and how to obtain the download media.

Checking the Product Interoperability Matrix

In this section, we will check to make sure the version of NSX we are deploying is compatible with the other VMware solutions in our environment.

  1. Navigate your web browser to the VMware product interoperability matrix webpage (http://www.vmware.com/go/interop)
  2. Select your vSphere solution as the first solution
  3. Add VMware NSX for vSphere as your second solution
  1. Add any other solutions that are specific to your environment
  2. Ensure all solution versions are compatible with one another before proceeding to download the NSX installation media

The following screenshot shows the official VMware product interoperability matrices that should be referenced before downloading NSX for vSphere:

Downloading media via the VMware downloads website

In this section we will download the installation media from the VMware downloads website as follows:

  1. From your web browser, navigate to the VMware downloads website (https://my.vmware.com/web/vmware/downloads).
  2. Scroll down to the Networking & Security menu item and click on Download Product
  1. Click on go to Downloads against your licensed tier for VMware NSX for vSphere 6.3.1 or whichever version is compatible with your environment
  2. Click on Download Now


Once you have downloaded the NSX for vSphere OVA, it is best practice to verify the file against the checksum listed to ensure that the downloaded file is an identical copy of the source.

Downloading media via the VMware Software Manager

In this section, we will download the VMware NSX installation media using the VMware Software Manager, in contrast to a manual download via the downloads website:

  1. Open the VMware Download Service application:

  1. Click on the VMware vSphere software suite
  2. Select VMware vSphere 6.5
  3. Select the licensing tier of your vSphere environment
  4. On the VMware NSX for vSphere menu pane, select the download button:

See also

To make sure that your vSphere and NSX version is supported by VMware, check the VMware life cycle product matrix (http://www.vmware.com/go/lifecycle). This list contains a list of unsupported products as well.


Deploying the NSX Manager virtual appliance

Deploying the NSX Manager virtual appliance is the first step to enabling network virtualization in your vSphere environment. In this recipe, you will go through the steps to enable your environment for NSX.

The following diagram depicts the logical process of enabling your environment for network virtualization, and the first four steps will be covered in this chapter:

Getting ready

Before deploying NSX Manager, the following prerequisites need to be satisfied:

  • Static IP address and portgroup for NSX Manager
  • Firewall ports open between NSX Manager, vCenter server, and ESXi VMKernel 0 Interface on each host (refer to Appendix for a complete list of ports)
  • Forward and reverse DNS entries for NSX Manager
  • NTP server is accessible; minimum of four is recommended for accurate time
  • Shared datastore for the appliance to be deployed onto
  • Satisfy minimum requirements for NSX Manager
  • Fill in the following table before deployment (removing prefilled data to reflect your environment):



NSX appliance name


NSX Manager hostname


vSphere cluster




vSphere network (Portgroup)

VM Network

IPv4 address

Subnet mask

Default gateway

Domain name


DNS server(s)

NTP servers(s) (Use four in production)

Enable SSH


CLI password


CLI privilege password


How to do it...

The following steps will detail how to deploy the NSX Manager appliance:

  1. Log into the vSphere Web Client
  2. Select Hosts and Clusters, right-click on the target cluster and select Deploy OVF Template
  1. Select Local File and locate the NSX Manager OVA downloaded earlier; click on Next
  2. Type in the Name of the virtual appliance and click on Next
  3. Select the vSphere cluster and resource where you want to deploy NSX Manager and select Next
  4. Review details, Accept license agreements and click on Next
  5. Select the shared datastore of where you want the virtual appliance to be deployment onto
  6. Select the VLAN-backed portgroup as defined earlier and click on Next
  7. Fill in the template details as highlighted in the preceding table and click on Next
  8. Ensure all details are correct and click on Finish:

Replacing the NSX Manager certificate

When you first deploy the NSX Manager, it creates a self-signed certificate. Using a self-signed certificate is generally not a recommended security practice. It is recommended to deploy a signed certificate from your internal certificate authority. NSX Manager supports two ways of deploying a signed certificate, which are as follows:

  • Certificate signing request to a Certificate Authority (CA)
  • Importing a PKCS#12 certificate archive (bundle) onto the NSX Manager, which includes the private and public key for NSX Manager and certificate chain of any subordinate CAs in your environment

In the following recipes, we will explore how you can create a certificate signing request on NSX Manager and how to import a PKCS#12 certificate bundle onto the NSX Manager.

Certificate Signing Request

A Certificate Signing Request (CSR) is the first part in a three-step process; this process involves the following steps:

  1. The NSX Manager creating a CSR
  2. The CSR is sent as a request to the certificate authority, which then signs the certificate and sends back a signed certificate
  3. Importing the signed certificate into the NSX Manager

How to do it...

The procedure to complete a certificate signing request is as follows:

  1. Log into NSX Manager via your web browser
  2. Click on Manage Appliance Settings
  3. Click on SSL Certificates
  1. Click on Generate CSR and follow the prompts as per the following screenshot:
  1. Click on OK and select Download CSR
  2. Send the CSR file to your security administrator and get the certificate signed
  3. With the returned certificate, click on Import so you can import the correct certificate into the NSX Manager
  4. Reboot the NSX Manager to complete the process of importing a signed certificate into the NSX Manager

PKCS#12 certificate

Importing PKCS#12 into the NSX Manager is used when the certificate signing was not completed using the CSR method outlined in the previous recipe. The PKCS#12 format is typically used in scripted installations of NSX Manager and other components. If a CSR was not generated by the NSX Manager itself, it is required that the PKCS#12 archive is imported into NSX Manager.

The PKCS#12 archive generally consists of the following:

  • A signed server certificate
  • A private key for the signed certificate
  • Root and intermediate certificate authority public keys

The PKCS#12 is also password-protected, so it's important to have the password before attempting to import the PKCS#12 archive into NSX Manager.

In some cases, the received signed certificate may not be in the PCKS#12 format. In this event, you must convert the certificates into the PKCS#12 format for import into the NSX Manager. This can be achieved using openSSL (https://www.openssl.org/), and the command to achieve this is as follows:

openssl pkcs12 -export -out server.p12 -inkey server.key -in server.crt -certfile CACert.crt

How to do it...

The procedure to import the PCKS#12 archive is as follows:

  1. Log into the NSX Manager via your web browser
  2. Click on Manage Appliance Settings
  3. Click on SSL Certificates
  4. Click on Upload PCKS#12Keystore and browse to the file
  5. Enter the password for archive and click on Import
  6. Reboot the NSX Manager to complete the process of importing the signed certificate

Registering vCenter server with NSX Manager

Once the NSX Manager appliance has been deployed and is accessible via https://nsxmgr-01a.corp.local, the next step is to register the NSX Manager as a solution against your vCenter deployment. NSX Manager and a vCenter server have a 1:1 relationship, and it's important to ensure that no other NSX Manager has previously been registered.

Getting ready

The following are things you need to consider before pairing the NSX Manager with the vCenter server:

  • Solution interoperability has been verified
  • vCenter server and vSphere environment are in a healthy state
  • Platform Services Controller (PSC) Fully Qualified Domain Name (FQDN) can be resolved
  • vCenter server FQDN can be resolved
  • vCenter and PSC time settings are verified
  • A service account with administrator role in vCenter has been created for the NSX Manager to use for registration; for further information refer to Chapter 9, Managing User Accounts in VMware NSX
  • TCP port 443 connectivity is required from the NSX Manager to the platform services controller and the vCenter server


vCenter server and platform services controller high availability options have been consulted to ensure the vCenter and PSC environment are set up as per VMware recommendations. For further information on supported vCenter high availability options, refer to the VMware KB article 1024051 (https://kb.vmware.com/kb/1024051).

How to do it...

The following section describes the steps to integrate NSX Manager with the vCenter server and the platform services controller, which are the first steps in enabling your environment for NSX.

Registering the NSX Manager with the vCenter server

The following are the steps to pair the NSX Manager with the vCenter server:

  1. Log into the NSX Manager web administration page: https://nsxmgr-01a.corp.local
  2. Navigate to Manage | NSX Management Services, and on the Lookup Service URL click on Edit
  1. Type the Lookup Server Host as the PSC FQDN or vCenter Server FQDN if using an embedded PSC
    1. For SSO Administrator Use Name, use the service account credentials defined
    2. Click on OKto complete
    3. When presented with the Trust Certificate dialog box, verify the SSL certificate thumbprint and click on Yes:


Modify Plugin Script download locationThis should only be modified if the NSX Manager is behind a firewall or "NAT" device which is masking the original IP address of the NSX Manager; in typical deployments, it will not require modification.

Registering the NSX Manager with the PSC

In this section we will register the NSX Manager with the Platform Services Controller for Single Sign-On services:

  1. Navigate back to the NSX management service web page on the NSX Manager web administration page: https://nsxmgr-01a.corp.local
  2. On the vCenter Server menu, click on Edit:
    1. Type the vCenter Server FQDN
    2. Type the service account credentials for the vCenter Service account and click on OK:
    1. When presented with the Trust Certificate dialog box, verify the SSL certificate thumbprint and click onYes

How it works...

The NSX Manager registers the com.vmware extension. This extension is installed on the vSphere web server as a plugin. When the plugin is installed onto the vSphere web server, any users that were logged in during integration will need to log out of the vSphere Web Client before they can start to consume the Networking & Security interface.


It is important to note that the account used from the NSX Manager to connect to vCenter server will be given enterprise administrator credentials. The NSX Manager uses the vSphere API to perform functions such as deploying service virtual machines, instructing the EAM service to prepare ESXi hosts, creating distributed portgroups, and other vSphere operations that it needs for NSX operations.

There's more...

If the event registration fails with the platform services controller, check the following commons issues first:

  • NTP Synchronization (time) for NSX Manager, platform services controller, and vCenter server is correct and aligned
  • DNS resolution for all components
  • Firewall ports are open if the NSX Manager and the PSC/vCenter server are separated in different security zones

Applying the NSX license

As described in choosing the right VMware NSX for vSphere edition, this section will describe the process of applying the license you have procured to utilize the features of NSX.

Getting ready

Things to verify before applying the NSX for vSphere license:

  • Correct license procured for installation of NSX
  • NSX has been integrated as a solution with your vSphere deployment

How to do it...

Perform the following steps to apply the NSX license to your installation:

  1. Log into the vSphere Web Client and click on Administration
  2. Click on Licenses under the Licensing section on the sidebar
  3. Select the Licenses tab and click on the plus sign:
    1. Enter your license key and click on Next
    2. Create a descriptive name for your license and click on Finish
  1. Next, select the Solutions tab and select the NSX Installation:
    1. Navigate to Actions|Assign License
    2. Select the license you added earlier and click on OK

Deploying the NSX Controller Cluster

The NSX controller cluster is an integral part of any NSX for vSphere deployment; the NSX controller cluster is responsible for:

  • Managing the vSphere hypervisor routing and switching modules
  • Managing the ARP table, MAC table, and VXLAN network identifier (VNI) information of the entire vSphere for NSX deployment
  • Distributed Logical Router:
    • Interfaces
    • Layer 2 Bridging Tables
    • Routes


The NSX Controller Cluster is the control plane for all networking constructs in an NSX deployment, however, the Distributed Firewall control plane is managed by the NSX Manager itself.

Getting ready

The following are things to consider before deploying the NSX controller cluster:

  • The controller cluster has three controllers in total and must be deployed in a cluster of three.
  • Each controller node should reside on a separate ESXi host; DRS anti-affinity rules should be used to enforce this rule. It is generally recommended to deploy controllers on a vSphere cluster with a minimum of four ESXi hosts.
  • Sufficient resources (vCPU, memory, and storage) on the vSphere cluster.
  • NSX controller nodes should be deployed onto shared storage that is highly available.
  • Each NSX controller requires an IPv4 address; these addresses are allocated via the NSX IP pool construct.
  • NSX controllers require connectivity to NSX Manager and vSphere management VMKernel IP addresses.
  • NSX controller should reside on a VLAN-backed PortGroup.

The NSX Controller IP Pool requires the following details prior to configuration. You can change values to suit your environment:






Prefix Length


Primary DNS

Secondary DNS

DNS Suffix


Static IP Pool -

How to do it...

In the following sub-sections, we proceed to start deploying the NSX controller cluster, which is required for the logical networking components in NSX.

Configuring an NSX IP pool

Before deploying the NSX controller cluster, an IP pool must be configured to reserve three IPv4 addresses on the network:

  1. In the vCenter Web Client, navigate to Networking & Security | NSX Managers | NSX Manager
  2. Select IP Pools and click on the plus sign
  3. Fill in the details as per the preceding table and click on OK

The IP Pool can be configured during deployment of the NSX controller cluster as well in the event it is not configured beforehand.

NSX Controller Cluster deployment

In this section, we will deploy each of three NSX Controllers on our vSphere cluster:

  1. In the vCenter Web Client, navigate to Networking & Security | Installation
  2. Under the NSX controller nodes menu pane, click on the plus sign
  3. Fill in the NSX controller details for the first node as follows and then click on OK:







Cluster/Resource Pool








Connected To




  1. After the first controller is deployed, repeat steps 1 to 3 for the remaining two controllers

Once all the controllers have been deployed, you should see the following displayed under the Installation tab in Networking & Security, with the green boxes indicating healthy connectivity between each of the peers in the controller cluster:

DRS Anti-Affinity Rules

DRS anti-affinity rules are required to ensure that the NSX controllers do not reside on the same physical host and are kept separate on dedicated ESXi hosts. This is to ensure in the event a ESXi host goes down where all three controllers are potentially running as guest VMs, the entire control plane for logical networking is not lost. If two controllers are lost, then the remaining controller goes into read-only mode until a cluster majority is restored.

It's important to note that the underlying infrastructure should still be designed for HA and resiliency, which includes compute/network/storage.

Configuring DRS anti-affinity rules via the vSphere web client:

  1. In the vCenter Web Client, navigate to Hosts and Clusters | Management Cluster | Manage | Settings | VM/Host Rules
  2. Click on Add...
  3. Choose Type as Separate Virtual Machines
  1. Add NSX controller virtual machines and click on OK:
Configuring DRS Anti-Affinity Rules via PowerCLI

You can also configure DRS anti-affinity rules using PowerCLI. To configure via PowerCLI, you will need to ensure PowerCLI has been deployed and installed locally on your system. Perform the following steps to configure the DRS rules via PowerCLI:

  1. Open the PowerCLI terminal up.
  2. Type Connect-VIServer -Server VCENTER_SERVER, which will connect your PowerCLI session to the vCenter server you are working on.
  1. Next, we want to retrieve the NSX controller virtual machines and store it as a variable, $nsx_controllers, using the get-vm PowerCLI cmdlet. The following code snippet demonstrates the command:
$nsx_controllers = get-vm | ? {$_.name -like "NSX_Controller*"}
  1. Next, using the New-DRSRule cmdlet, we will configure the anti-affinity DRS rule on the RegionA01-MGMT01 vSphere cluster using the following command:
New-DrsRule -Name nsx-controller-anti-affinity -Cluster RegionA01-MGMT01 -KeepTogether $false -VM $antiAffinityVMs

There's more...

In the following sub-sections, placement of the NSX Controllers and Controller password configuration will be discussed in greater detail.

Separate vCenter environment

The controller cluster is deployed in a group of three. Each controller node can only be deployed onto a vSphere cluster that is part of the vCenter inventory that the NSX Manager you are configuring is paired with. In large environments with multiple vCenters, it is not uncommon for the vCenter server and NSX Manager to be deployed onto a dedicated vSphere cluster that is managed by an independent vCenter server that is deemed as management. In this scenario, the NSX controller cluster cannot be deployed onto the dedicated management vSphere cluster.

Controller password parameters

The NSX controller password must meet the following criteria:

  • It must not contain the username as a substring
  • A character must not be repeated consecutively more than three times
  • It must be at least 12 characters long and must follow three of the following four rules:
    • It must have at least one uppercase letter
    • It must have at least one lowercase letter
    • It must have at least one number
    • It must have at least one special character

In the event that the NSX controller password is forgotten, it can be easily changed using the following steps:

  1. Log into the vSphere Web Client
  2. Click on the Networking & Security tab and then navigate to Installation | Management:
    1. Under the NSX Controller nodesmenu, selectActions
    2. Click on Change Controller Cluster Password
    3. Type a new password following the preceding guidelines and click on OK:

Preparing a vSphere cluster for NSX

Preparing a vSphere cluster for NSX does two things:

  1. It installs NSX Kernel modules on each ESXi host, which is a member of the vSphere cluster
  2. It builds the NSX control-plane and management-plane fabric

NSX Kernel modules are packaged as VMware installation bundles (VIBs) and provide functionality such as distributed routing, distributed firewall, and VXLAN bridging.

Getting ready

To get ready for installation, ensure that the following prerequisite tasks have been completed:

  • DNS forward and reverse names have been created for all ESXi hosts and are resolvable
  • Firewall Ports between all management components are open
  • vCenter Update Manager Service, if in use, is operational
    • Ensure that the EAM service is operational
    • Ensure that the NTP settings are checked across all ESXi hosts and are updating time correctly


ESXi stateless modeIf you are using ESXi in stateless mode, you must download the NSX VIBs manually and integrate them into the host image. Refer to VMware Knowledge Base Article 2041972 (https://kb.vmware.com/kb/2041972) for more information. Download paths of NSX VIBs change with each release. To check the paths for your NSX release, use the following URL: https://<NSX_MANAGER_IP>/bin/vdn/nwfabric.properties.

How to do it...

Perform the following steps to start the installation of the NSX VIBs onto your first vSphere cluster; we will be enabling it on vSphere Cluster RegionA01-COMP01 to begin with:

  1. In the vCenter Web Client, navigate to Networking & Security | Installation | Host Preparation
  2. Select the vSphere Cluster RegionA01-COMP01
  3. Click on the COG wheel and select Install:

Each ESXi host in the cluster will now download the VIBs from vCenter Server, where they were downloaded from NSX Manager and cached when NSX was registered as a solution. Depending on the number of hosts in the vSphere cluster, this process will take a few minutes to complete. Once the installation has completed, you will be presented with a screen like the one shown in the following screenshot:

How it works...

The following figure depicts the management, control, and data plane components that make up an NSX implementation. Each has an important part to play in enabling ESXi for the Distributed Firewall and VXLAN. In this section, we will explore the interaction among the various components:

  • vCenter server: This is the management component of the vSphere environment and is where the networking and security components of an NSX environment are all managed from.
  • NSX Manager: This is the management plane of the NSX implementation. It integrates directly with vCenter and manages both the NSX controller cluster and the ESXi hosts. The NSX Manager is also responsible for pushing distributed firewall rules to each host that is prepared for the distributed firewall. In addition, the NSX Manager is also the API entry point for NSX operations via the REST protocol.
  • ESXi Agency Manager (EAM): This is part of the vCenter deployment; it is responsible for installing the VIBs to each of the hosts.

When you prepare a vSphere cluster for NSX, the VIBs are copied directly from NSX Manager and cached by EAM. The EAM will then track the installation of each VIB onto each host in the vSphere cluster. If the VIB is not present, it is installed without the ESXi host requiring a reboot, and if it is present, a reboot is required to complete the upgrade.

Once the installation of VIBs has been completed, each ESXi host will have active TCP connections to the NSX Manager and NSX controller cluster. The connection to the NSX Manager is from the vsfwd daemon running on the ESXi host via the RabbitMQ message bus. The connection to the NSX Controller cluster is from the netcpa daemon running on the ESXi host via an SSL connection (TCP Port 1234). It is important that both channels of communication are active and can be checked via the communication channel health from each host, which is covered in a subsequent section:

Enabling NSX in a brownfield environment

When enabling a vSphere cluster for NSX in a brownfield environment, it is important to be cognizant that any preconfigured DFW firewall rules have the potential to impact virtual machines on the newly-configured vSphere cluster. Therefore, it is extremely important to ensure that the default Distributed Firewall rule remains as allow any any. Changing to deny before defining rules for allowing legitimate traffic from/to virtual machines will cause traffic blackholing. 

As a best practice, vCenter server and virtual machines that require promiscuous mode should be excluded from the DFW if you are not planning to protect them. To learn how to exclude virtual machines from the DFW, refer to Chapter 6Configuring VMware NSX Distributed Firewall (DFW) and SpoofGuard.


Validating NSX VIB installation

Installation of NSX VIBs that enable the Distributed Firewall and VXLAN are essential for a working NSX environment. This section will investigate how to manually verify that each VIB is installed correctly and whether communication to both the NSX controller cluster and NSX Manager are present.

Distributed Firewall communication

The first control plane communication that we are concerned with is from the NSX Manager to each ESXi host via TCP port 5671. This port is reserved for the Rabbit MQ Message bus to the vsfwd daemon running on each host after the VMware Service Insertion Platform (VSIP) VIB installation, which is the Distributed Firewall kernel module. The NSX Manager uses the message bus to publish firewall rules down to each ESXi host. The ESXi host then applies them to vNICs of virtual machines that are running on top of its hypervisor.

Controller communication

The second control plane communication that is expected from each ESXi host is an open connection to each of the NSX controllers deployed. The NSX controller cluster is responsible for control plane information for ARP/MAC/VTEP tables. It is also used to program routes received on the Distributed Logical Router Control VM to each host (more on this in Chapter 2, Configuring VMware NSX Logical Switch Networks). From each host, we expect the netcpa daemon to have an active connection to the controller cluster on TCP port 1234.

Getting ready

To manually verify control-plane communication and VIB installation, you will need the following access to the following NSX components:

  • SSH access to NSX Manager
  • SSH access to each NSX controller
  • SSH access to ESXi hosts that were prepared for NSX

You would not be expected to check communication of each and every host in your environment, as this can become unwieldly. However, this section is included for you to understand what the expected communication is, but in large deployments you would check the communication channel health per vSphere cluster as depicted in the earlier section.

How to do it...

To check whether the NSX VIBs have been installed successfully is crucial. The upcoming section details how to do this manually on an ESXi host and how to check NSX component communication.

Manually checking VIB installation

In this section we perform manual verification that the VIBs have been successfully installed.

  1. SSH onto an ESXi host.
  2. Check whether VXLAN VIB modules have been installed by executing the following command:
esxcli software vib get --vibname esx-vxlan
  1. You will receive an output similar to the following:
[root@vSphere:~]    esxcli software vib get --vibname esx-vxlanVMware_bootbank_esx-vxlan_6.0.0-0.0.4987429   Name: esx-vxlan   Version: 6.0.0-0.0.4987429Type: bootbankVendor: VMwareAcceptance Level: VMwareCertifiedSummary: Vxlan and host toolDescription: This package loads module and configures firewall for vxlan networking. ReferenceURLs:Creation Date: 2017-01-27Depends: esx-base >= 6.0, esx-base <= 6.5.0, nsx-api <= 2.1, vmkapi_2_3_0_0Conflicts: nsx-api = 2Replaces: esx-traceflow, esx-dvfilter-switch-security, esx-bfdProvides: com.vmware.vxlan =, com.vmware.switchsecurity =, com.vmware.traceflow =, com.vmware.bfd = Mode Required: TrueHardware Platforms Required:Live Install Allowed: TrueLive Remove Allowed: TrueStateless Ready: TrueOverlay: FalseTags:Payloads: esx-vxlan
  1. If the module has been installed correctly, you should see open TCP connections on port 1234 with the following command:
esxcli network ip connection list | grep 1234

An example is included below that shows the connection as established to each of the three NSX controllers from the point of view of an ESXi host:

  1. To see which NSX controllers the host is configured to communicate with, execute the following command:
cat /etc/vmware/netcpa/config-by-vsm.xml

The following screenshot provides a truncated output of the command and its expected output.

  1. Check if VSIP VIB modules have been installed by executing the following command: esxcli software vib get --vibname esx-vsip:

  1. You will receive an output similar to the following:
[root@vSphere:~] esxcli software vib get --vibname esx-vsipVMware_bootbank_esx-vsip_6.0.0-0.0.4987429   Name: esx-vsipVersion: 6.0.0-0.0.4987429Type: bootbankVendor: VMwareAcceptance Level: VMwareCertifiedSummary: vsip moduleDescription: This package contains DFW and NetX data and control plane components.ReferenceURLs:Creation Date: 2017-01-27Depends: esx-base >= 6.0, esx-base <= 6.5.0, nsx-api <= 2.1, vmkapi_2_3_0_0Conflicts: nsx-api = 2Replaces: esx-vdpi Provides: vsip = 1.0.0-0Maintenance Mode Required: TrueHardware Platforms Required:Live Install Allowed: TrueLive Remove Allowed: TrueStateless Ready: TrueOverlay: FalseTags:Payloads: esx-vsip
  1. If the module has been installed correctly, you can open TCP connections on port 5671 with the following command:
esxcli network ip connection list | grep 5671

The following screenshot provides an example of the above command, the output shows a connection of established to the NSX Manager over TCP Port 5671:

  1. To see which NSX Manager the host is configured to communicate with, execute the following command:
esxcfg-advcfg -g /UserVars/RmqIpAddress

The following screenshot shows the results of the execution of the command above, and the expected configuration parameter is the IP address of the NSX Manager:

Checking NSX component communication

To check the communication channel between NSX Manager, NSX controller cluster, control plane agent, and Distributed Firewall agent, follow the following procedure:

  1. In the vCenter Web Client, navigate to Networking & Security | Installation | Host Preparation
  2. Select your vSphere cluster or an ESXi host
  1. Click on the Actions button and navigate to | Communication Channel Health:

The following screenshot displays the result of the preceding action and the ESXi communication health:

About the Authors
  • Bayu Wibowo

    Bayu Wibowo is a seasoned network virtualization consultant in the APJ arena. With over 10 years of industry experience, he has rapidly earned reputation and awards for his community involvement as Cisco Champion, VMware vExpert NSX, and VMTN Community Warrior. Working as a network virtualization consultant for Datacom, he now plays an integral part in the implementation of multiple innovative technologies, including VMware NSX, Open Networking, and numerous more. Follow him on Twitter @bayupw.

    Browse publications by this author
  • Tony Sangha

    Tony Sangha is a senior consulting architect at VMware Professional Services with over 12 years of experience in networking and security roles, who has worked for a systems integrator across various industry verticals. He guides customers across Australia and New Zealand to design and implement a Software Defined Datacenter using VMware technologies and specializes in VMware NSX. He has presented at multiple VMUG and vForum events across ANZ and is an active community contributor via his blog and open source projects on GitHub. You can follow him on Twitter at @tsangha.

    Browse publications by this author
Latest Reviews (1 reviews total)
Excellent book that gives insight into how to utilize NSX.
Recommended For You
VMware NSX Cookbook
Unlock this book and the full library FREE for 7 days
Start now