Download code from GitHub
In this chapter, we will explore how to install and configure NSX for vSphere. We will be covering the following recipes:
- Choosing the right VMware NSX for vSphere edition
- Selecting ESXi hosts and network adapters
- Downloading NSX for vSphere
- Deploying the NSX Manager virtual appliance
- Replacing the NSX Manager certificate
- Registering vCenter server with NSX Manager
- Applying the NSX licenses
- Deploying the NSX Controller Cluster
- Preparing a vSphere cluster for NSX
- Validating NSX VIB installation
- Checking NSX component communication
This book aims to be useful for both new and seasoned VMware NSX for vSphere administrators. It is intended to be used by those that have never deployed NSX and by those that have it deployed already but are looking to leverage newer or advanced functionality. Intermediate networking and virtualization knowledge is assumed and is essential to understanding deployment of NSX into your environment.
Before we begin serving the main recipes of our cookbook, we will first provide an overview of what VMware NSX for vSphere is and what functionality it provides over traditional networking models.
VMware NSX for vSphere is a core component of the VMware Software-Defined Data Center (SDDC); it is the component that enables network virtualization. Network virtualization provides a layer of abstraction over the physical network using a VXLAN network overlay. With NSX, network operations are now independent of the physical hardware, and functions such as logical firewalls, load balancers, logical routers, logical switches, and virtual private networks can be provisioned, modified, or torn down as part of an automated workflow.
VMware NSX has four licensing editions: standard, advanced, enterprise, and remote office/branch offices (ROBO). Each licensing tier provides distinctive functionality, available per CPU socket on a perpetual basis at the vSphere cluster level.
The standard and advanced editions are also available as per 100 users in a pack basis to align with virtual desktop deployments (vSphere for desktop). The enterprise edition is also available on per-VM term basis. You can upgrade from standard to advanced/enterprise and from advanced to enterprise.
Note
Prior to NSX 6.2.2, VMware NSX for vSphere did not have multiple licensing tiers. If you purchased NSX prior to May 3, 2016, you are entitled to VMware NSX Enterprise edition as long as you have active support and subscription contracts. You can upgrade your VMware NSX license key from the My VMware portal (http://my.vmware.com).
Like vSphere licensing, VMware NSX is licensed per CPU socket. If you have a separate Management vSphere Cluster that is used for Infrastructure VMs and are not planning to protect it with the NSX Distributed Firewall or place NSX Edge Service Gateways onto it, you are not required to license the CPUs on that Management vSphere Cluster. The Compute vSphere cluster and Edge vSphere cluster need to be licensed.
Note
VMware NSX is licensed at the vSphere Cluster level. If you need to exclude a specific ESXi host from NSX, you will need to remove the ESXi host from the cluster. For vSphere environments with VMware vCenter Site Recovery Manager, you will normally have active sites (Protected site) and passive/disaster recovery sites (Recovery site). Only the active ESXi hosts on the protected site requires a VMware NSX license. For more about licensing NSX for vSphere see VMware KB 2078615 (https://kb.vmware.com/kb/2078615).
From your vSphere inventory you will need to do the following:
- Determine how many CPU sockets you need
- Determine the NSX features required
- If you are planning to integrate third-party partner solutions with NSX (http://www.vmware.com/products/nsx/technology-partners.html), check whether a specific NSX feature is required
Note
Some security services partner solutions require NSX distributed firewalling features and physical-to-virtual data center services requires integration with a Hardware VTEP (HW VTEP).
- Choose the NSX edition based on the features required
Note
I would like to use VMware vShield Endpoint for anti-virus/anti-malware capability only. Which NSX edition should I use? VMware vShield endpoint is included as a vSphere feature in the vSphere Essential Plus Edition or later, so you do not need to purchase VMware a NSX license. VMware NSX for vShield endpoint will appear on the vSphere download site if you have vSphere Essential Plus Edition or later. For more information, see VMware KB 2110078 (https://kb.vmware.com/kb/2110078).
The following sub-sections will detail the different tiers of NSX licensing and the features available in each. From there, how to evaluate and purchase VMware NSX will also be detailed.
The four tiers of licenses are as follows:
- Standard edition
- Advanced edition
- Enterprise edition
- ROBO
The features available in each edition are as follows:
Product feature | Standard | Advanced | Enterprise | ROBO |
Distributed Switching | ● | ● | ● | ● |
Distributed Routing | ● | ● | ||
NSX Edge Firewall | ● | ● | ● | ● |
Network Address Translation (NAT) | ● | ● | ● | ● |
SW L2 Bridging to physical environment | ● | ● | ● | |
Dynamic routing with ECMP (Active-Active) | ● | ● | ● | ● |
API-driven | ● | ● | ● | ● |
Integration with vRealize and OpenStack | ● | ● | ● | ● |
Automation of security policies with vRealize | ● | ● | ● | |
NSX Edge Load Balancing | ● | ● | ● | |
Distributed Firewalling | ● | ● | ● | |
Integration with Active Directory | ● | ● | ● | |
Service Insertion (third-party integration) | ● | ● | ● | |
Cross vCenter NSX | ● | |||
Multisite NSX optimizations | ● | |||
VPN (IPSec and SSL) | ● | ● | ||
Remote gateway | ● | |||
Integration with HW VTEPs | ● |
There are two ways to evaluate VMware products:
Note
VMware NSX license is not available publicly. Contact your VMware sales representative to get an NSX evaluation license.
- Use VMware Hands-on Labs (http://labs.hol.vmware.com/) to experience VMware NSX in a virtual lab environment:
- VMware NSX Hands-on Lab Intro (http://www.vmware.com/go/try-nsx-en)
- VMware NSX Hands-on Lab Advanced (http://www.vmware.com/go/try-nsx-adv-hol)
There are support and subscription plan options that you can purchase in addition to the product:
- Basic support: 12 hours a day technical support during business hours
- Production support: 24 hours (Severity 1), seven days a week support
The production support plan is recommended for production and critical environments. If you need higher-level support above production grade, additional support options such as Business Critical Support (BCS) or Mission Critical Support (MCS) can be purchased on top of production support. For more information on VMware support offerings, see https://www.vmware.com/support/services.html.
VMware vRealize Log Insight is a log management engine that collects logs from a number of different sources and provides rich dashboards and search functionality.
Log Insight is available for NSX at no additional charge, you are entitled to one Log Insight CPU per NSX CPU license. The support and subscription is included with the NSX purchase. It is a fully functioning version of Log Insight but limited to vSphere and NSX data sources and content packs only. If you need more data sources and content packs, additional Log Insight licenses are required.
For more information about the VMware NSX Neutron plugin license editions for VMware integrated OpenStack, see VMware KB 2145269 (https://kb.vmware.com/kb/2145269).
Similar to the requirements of a VMware vSphere solution, choosing the correct hardware is still an important part of any NSX deployment; therefore, you need to follow the same process that you did for vSphere to ensure the hardware you are deploying is on the VMware Compatibility Guide (http://www.vmware.com/resources/compatibility/search.php).
The compatibility guide does not only list the supported servers, but you need to also check if your network interface card (I/O devices) is supported and features such as VXLAN Offload and Receive Side Scaling are also supported.
VXLAN Offload is akin to TCP segmentation offload (TSO), but compared to TSO, which is designed for TCP packet headers, VXLAN encapsulates the original (source) packet from a virtual machine into a user datagram protocol (UDP) packet with its own unique header, known as the VXLAN header. Placing this additional header onto a packet invalidates traditional offloading mechanisms in-place and therefore increases load on the CPU as additional CPU cycles are needed to encapsulate and decapsulate every VXLAN packet. VXLAN is covered in greater detail in Chapter 2,Configuring VMware NSX Logical Switch Networks.
In this recipe, we will download the installation media for NSX for vSphere. The installation media comes in the form of an open virtual application (OVA) that is distributed through the VMware downloads site (https://my.vmware.com/web/vmware/downloads).
To download NSX for vSphere, the following prerequisites must be satisfied:
- Valid VMware software entitlements that enable you to download the installation media
- Access to the VMware downloads website
- Access to VMware software manager. Download and install VMware software manager first (https://www.vmware.com/go/download-software-manager-en)
- VMware product interoperability matrix has been consulted so you know which version is compatible with your environment
The following sections will explain how to check that your infrastructure supports the version of NSX you are implementing and how to obtain the download media.
In this section, we will check to make sure the version of NSX we are deploying is compatible with the other VMware solutions in our environment.
- Navigate your web browser to the VMware product interoperability matrix webpage (http://www.vmware.com/go/interop)
- Select your vSphere solution as the first solution
- Add VMware NSX for vSphere as your second solution
- Add any other solutions that are specific to your environment
- Ensure all solution versions are compatible with one another before proceeding to download the NSX installation media
The following screenshot shows the official VMware product interoperability matrices that should be referenced before downloading NSX for vSphere:
In this section we will download the installation media from the VMware downloads website as follows:
- From your web browser, navigate to the VMware downloads website (https://my.vmware.com/web/vmware/downloads).
- Scroll down to the
Networking & Securitymenu item and click onDownload Product
- Click on go to
Downloadsagainst your licensed tier for VMware NSX for vSphere 6.3.1 or whichever version is compatible with your environment - Click on
Download Now
In this section, we will download the VMware NSX installation media using the VMware Software Manager, in contrast to a manual download via the downloads website:
- Open the
VMwareDownload Serviceapplication:
- Click on the
VMware vSpheresoftware suite - Select
VMware vSphere 6.5 - Select the licensing tier of your vSphere environment
- On the
VMware NSX for vSpheremenu pane, select the download button:
To make sure that your vSphere and NSX version is supported by VMware, check the VMware life cycle product matrix (http://www.vmware.com/go/lifecycle). This list contains a list of unsupported products as well.
Deploying the NSX Manager virtual appliance is the first step to enabling network virtualization in your vSphere environment. In this recipe, you will go through the steps to enable your environment for NSX.
The following diagram depicts the logical process of enabling your environment for network virtualization, and the first four steps will be covered in this chapter:
Before deploying NSX Manager, the following prerequisites need to be satisfied:
- Static IP address and portgroup for NSX Manager
- Firewall ports open between NSX Manager, vCenter server, and ESXi VMKernel 0 Interface on each host (refer to Appendix for a complete list of ports)
- Forward and reverse DNS entries for NSX Manager
- NTP server is accessible; minimum of four is recommended for accurate time
- Shared datastore for the appliance to be deployed onto
- Satisfy minimum requirements for NSX Manager
- Fill in the following table before deployment (removing prefilled data to reflect your environment):
Component | Value |
NSX appliance name |
|
NSX Manager hostname |
|
vSphere cluster |
|
Datastore |
|
vSphere network (Portgroup) |
|
IPv4 address |
|
Subnet mask |
|
Default gateway |
|
Domain name |
|
DNS server(s) |
|
NTP servers(s) |
|
Enable SSH |
|
CLI password |
|
CLI privilege password |
|
The following steps will detail how to deploy the NSX Manager appliance:
- Log into the vSphere Web Client
- Select
Hosts and Clusters, right-click on the target cluster and selectDeploy OVF Template
- Select
Local Fileand locate the NSX Manager OVA downloaded earlier; click onNext - Type in the
Nameof the virtual appliance and click onNext - Select the vSphere cluster and resource where you want to deploy NSX Manager and select
Next - Review details,
Acceptlicense agreements and click onNext - Select the shared datastore of where you want the virtual appliance to be deployment onto
- Select the VLAN-backed portgroup as defined earlier and click on
Next - Fill in the template details as highlighted in the preceding table and click on
Next - Ensure all details are correct and click on
Finish:
When you first deploy the NSX Manager, it creates a self-signed certificate. Using a self-signed certificate is generally not a recommended security practice. It is recommended to deploy a signed certificate from your internal certificate authority. NSX Manager supports two ways of deploying a signed certificate, which are as follows:
In the following recipes, we will explore how you can create a certificate signing request on NSX Manager and how to import a PKCS#12 certificate bundle onto the NSX Manager.
A Certificate Signing Request (CSR) is the first part in a three-step process; this process involves the following steps:
- The NSX Manager creating a CSR
- The CSR is sent as a request to the certificate authority, which then signs the certificate and sends back a signed certificate
- Importing the signed certificate into the NSX Manager
The procedure to complete a certificate signing request is as follows:
- Log into NSX Manager via your web browser
- Click on
Manage Appliance Settings - Click on
SSL Certificates
- Click on
Generate CSRand follow the prompts as per the following screenshot:
- Click on
OKand selectDownload CSR - Send the CSR file to your security administrator and get the certificate signed
- With the returned certificate, click on
Importso you can import the correct certificate into the NSX Manager - Reboot the NSX Manager to complete the process of importing a signed certificate into the NSX Manager
Importing PKCS#12 into the NSX Manager is used when the certificate signing was not completed using the CSR method outlined in the previous recipe. The PKCS#12 format is typically used in scripted installations of NSX Manager and other components. If a CSR was not generated by the NSX Manager itself, it is required that the PKCS#12 archive is imported into NSX Manager.
The PKCS#12 archive generally consists of the following:
- A signed server certificate
- A private key for the signed certificate
- Root and intermediate certificate authority public keys
The PKCS#12 is also password-protected, so it's important to have the password before attempting to import the PKCS#12 archive into NSX Manager.
In some cases, the received signed certificate may not be in the PCKS#12 format. In this event, you must convert the certificates into the PKCS#12 format for import into the NSX Manager. This can be achieved using openSSL (https://www.openssl.org/), and the command to achieve this is as follows:
openssl pkcs12 -export -out server.p12 -inkey server.key -in server.crt -certfile CACert.crtThe procedure to import the PCKS#12 archive is as follows:
- Log into the NSX Manager via your web browser
- Click on
Manage Appliance Settings - Click on
SSL Certificates - Click on
Upload PCKS#12Keystoreand browse to the file - Enter the password for archive and click on
Import - Reboot the NSX Manager to complete the process of importing the signed certificate
Once the NSX Manager appliance has been deployed and is accessible via https://nsxmgr-01a.corp.local, the next step is to register the NSX Manager as a solution against your vCenter deployment. NSX Manager and a vCenter server have a 1:1 relationship, and it's important to ensure that no other NSX Manager has previously been registered.
The following are things you need to consider before pairing the NSX Manager with the vCenter server:
- Solution interoperability has been verified
- vCenter server and vSphere environment are in a healthy state
- Platform Services Controller (PSC) Fully Qualified Domain Name (FQDN) can be resolved
- vCenter server FQDN can be resolved
- vCenter and PSC time settings are verified
- A service account with administrator role in vCenter has been created for the NSX Manager to use for registration; for further information refer to Chapter 9, Managing User Accounts in VMware NSX
- TCP port
443connectivity is required from the NSX Manager to the platform services controller and the vCenter server
Note
vCenter server and platform services controller high availability options have been consulted to ensure the vCenter and PSC environment are set up as per VMware recommendations. For further information on supported vCenter high availability options, refer to the VMware KB article 1024051 (https://kb.vmware.com/kb/1024051).
The following section describes the steps to integrate NSX Manager with the vCenter server and the platform services controller, which are the first steps in enabling your environment for NSX.
The following are the steps to pair the NSX Manager with the vCenter server:
- Log into the NSX Manager web administration page:
https://nsxmgr-01a.corp.local - Navigate to
Manage|NSX Management Services, and on theLookup Service URLclick onEdit
- Type the Lookup Server Host as the PSC FQDN or vCenter Server FQDN if using an embedded PSC
- For SSO Administrator
Use Name, use the service account credentials defined - Click on
OKto complete - When presented with the Trust Certificate dialog box, verify the SSL certificate thumbprint and click on
Yes:
- For SSO Administrator
In this section we will register the NSX Manager with the Platform Services Controller for Single Sign-On services:
- Navigate back to the NSX management service web page on the NSX Manager web administration page:
https://nsxmgr-01a.corp.local - On the vCenter Server menu, click on
Edit:- Type the vCenter Server FQDN
- Type the service account credentials for the vCenter Service account and click on
OK:
- When presented with the Trust Certificate dialog box, verify the SSL certificate thumbprint and click on
Yes
- When presented with the Trust Certificate dialog box, verify the SSL certificate thumbprint and click on
The NSX Manager registers the com.vmware extension. This extension is installed on the vSphere web server as a plugin. When the plugin is installed onto the vSphere web server, any users that were logged in during integration will need to log out of the vSphere Web Client before they can start to consume the Networking & Security interface.
Note
It is important to note that the account used from the NSX Manager to connect to vCenter server will be given enterprise administrator credentials. The NSX Manager uses the vSphere API to perform functions such as deploying service virtual machines, instructing the EAM service to prepare ESXi hosts, creating distributed portgroups, and other vSphere operations that it needs for NSX operations.
If the event registration fails with the platform services controller, check the following commons issues first:
- NTP Synchronization (time) for NSX Manager, platform services controller, and vCenter server is correct and aligned
- DNS resolution for all components
- Firewall ports are open if the NSX Manager and the PSC/vCenter server are separated in different security zones
As described in choosing the right VMware NSX for vSphere edition, this section will describe the process of applying the license you have procured to utilize the features of NSX.
Things to verify before applying the NSX for vSphere license:
- Correct license procured for installation of NSX
- NSX has been integrated as a solution with your vSphere deployment
Perform the following steps to apply the NSX license to your installation:
- Log into the vSphere Web Client and click on
Administration - Click on
Licensesunder the Licensing section on the sidebar - Select the
Licensestab and click on the plus sign:- Enter your license key and click on
Next - Create a descriptive name for your license and click on
Finish
- Enter your license key and click on
- Next, select the
Solutionstab and select the NSX Installation:- Navigate to
Actions|Assign License - Select the license you added earlier and click on
OK
- Navigate to
The NSX controller cluster is an integral part of any NSX for vSphere deployment; the NSX controller cluster is responsible for:
Note
The NSX Controller Cluster is the control plane for all networking constructs in an NSX deployment, however, the Distributed Firewall control plane is managed by the NSX Manager itself.
The following are things to consider before deploying the NSX controller cluster:
- The controller cluster has three controllers in total and must be deployed in a cluster of three.
- Each controller node should reside on a separate ESXi host; DRS anti-affinity rules should be used to enforce this rule. It is generally recommended to deploy controllers on a vSphere cluster with a minimum of four ESXi hosts.
- Sufficient resources (vCPU, memory, and storage) on the vSphere cluster.
- NSX controller nodes should be deployed onto shared storage that is highly available.
- Each NSX controller requires an IPv4 address; these addresses are allocated via the NSX IP pool construct.
- NSX controllers require connectivity to NSX Manager and vSphere management VMKernel IP addresses.
- NSX controller should reside on a VLAN-backed PortGroup.
The NSX Controller IP Pool requires the following details prior to configuration. You can change values to suit your environment:
Component | Value |
|
|
Gateway |
|
|
|
|
|
| |
|
|
|
|
In the following sub-sections, we proceed to start deploying the NSX controller cluster, which is required for the logical networking components in NSX.
Before deploying the NSX controller cluster, an IP pool must be configured to reserve three IPv4 addresses on the network:
- In the vCenter Web Client, navigate to
Networking & Security|NSX Managers|NSX Manager - Select
IP Poolsand click on the plus sign - Fill in the details as per the preceding table and click on
OK
The IP Pool can be configured during deployment of the NSX controller cluster as well in the event it is not configured beforehand.
In this section, we will deploy each of three NSX Controllers on our vSphere cluster:
- In the vCenter Web Client, navigate
to Networking & Security|Installation - Under the NSX controller nodes menu pane, click on the plus sign
- Fill in the NSX controller details for the first node as follows and then click on
OK:
Name | Value |
|
|
|
|
|
|
|
|
| Optional |
| Optional |
|
|
|
|
- After the first controller is deployed, repeat steps 1 to 3 for the remaining two controllers
Once all the controllers have been deployed, you should see the following displayed under the Installation tab in Networking & Security, with the green boxes indicating healthy connectivity between each of the peers in the controller cluster:
DRS anti-affinity rules are required to ensure that the NSX controllers do not reside on the same physical host and are kept separate on dedicated ESXi hosts. This is to ensure in the event a ESXi host goes down where all three controllers are potentially running as guest VMs, the entire control plane for logical networking is not lost. If two controllers are lost, then the remaining controller goes into read-only mode until a cluster majority is restored.
It's important to note that the underlying infrastructure should still be designed for HA and resiliency, which includes compute/network/storage.
Configuring DRS anti-affinity rules via the vSphere web client:
- In the vCenter Web Client, navigate to
Hosts and Clusters|Management Cluster|Manage|Settings|VM/Host Rules - Click on
Add... - Choose
TypeasSeparate Virtual Machines
- Add NSX controller virtual machines and click on
OK:
You can also configure DRS anti-affinity rules using PowerCLI. To configure via PowerCLI, you will need to ensure PowerCLI has been deployed and installed locally on your system. Perform the following steps to configure the DRS rules via PowerCLI:
- Open the PowerCLI terminal up.
- Type
Connect-VIServer -Server VCENTER_SERVER, which will connect your PowerCLI session to the vCenter server you are working on.
- Next, we want to retrieve the NSX controller virtual machines and store it as a variable,
$nsx_controllers, using theget-vmPowerCLI cmdlet. The following code snippet demonstrates the command:
$nsx_controllers = get-vm | ? {$_.name -like "NSX_Controller*"}- Next, using the
New-DRSRule cmdlet, we will configure the anti-affinity DRS rule on theRegionA01-MGMT01vSphere cluster using the following command:
New-DrsRule -Name nsx-controller-anti-affinity -Cluster RegionA01-MGMT01 -KeepTogether $false -VM $antiAffinityVMsIn the following sub-sections, placement of the NSX Controllers and Controller password configuration will be discussed in greater detail.
The controller cluster is deployed in a group of three. Each controller node can only be deployed onto a vSphere cluster that is part of the vCenter inventory that the NSX Manager you are configuring is paired with. In large environments with multiple vCenters, it is not uncommon for the vCenter server and NSX Manager to be deployed onto a dedicated vSphere cluster that is managed by an independent vCenter server that is deemed as management. In this scenario, the NSX controller cluster cannot be deployed onto the dedicated management vSphere cluster.
The NSX controller password must meet the following criteria:
- It must not contain the username as a substring
- A character must not be repeated consecutively more than three times
- It must be at least 12 characters long and must follow three of the following four rules:
- It must have at least one uppercase letter
- It must have at least one lowercase letter
- It must have at least one number
- It must have at least one special character
In the event that the NSX controller password is forgotten, it can be easily changed using the following steps:
- Log into the vSphere Web Client
- Click on the
Networking & Securitytab and then navigate toInstallation|Management:- Under the
NSX Controller nodesmenu, selectActions - Click on
Change Controller Cluster Password - Type a new password following the preceding guidelines and click on
OK:
- Under the
Preparing a vSphere cluster for NSX does two things:
- It installs NSX Kernel modules on each ESXi host, which is a member of the vSphere cluster
- It builds the NSX control-plane and management-plane fabric
NSX Kernel modules are packaged as VMware installation bundles (VIBs) and provide functionality such as distributed routing, distributed firewall, and VXLAN bridging.
To get ready for installation, ensure that the following prerequisite tasks have been completed:
- DNS forward and reverse names have been created for all ESXi hosts and are resolvable
- Firewall Ports between all management components are open
- vCenter Update Manager Service, if in use, is operational
- Ensure that the EAM service is operational
- Ensure that the NTP settings are checked across all ESXi hosts and are updating time correctly
Note
ESXi stateless modeIf you are using ESXi in stateless mode, you must download the NSX VIBs manually and integrate them into the host image. Refer to VMware Knowledge Base Article 2041972 (https://kb.vmware.com/kb/2041972) for more information. Download paths of NSX VIBs change with each release. To check the paths for your NSX release, use the following URL: https://<NSX_MANAGER_IP>/bin/vdn/nwfabric.properties.
Perform the following steps to start the installation of the NSX VIBs onto your first vSphere cluster; we will be enabling it on vSphere Cluster RegionA01-COMP01 to begin with:
- In the vCenter Web Client, navigate to
Networking & Security|Installation|Host Preparation - Select the vSphere Cluster
RegionA01-COMP01 - Click on the COG wheel and select
Install:
Each ESXi host in the cluster will now download the VIBs from vCenter Server, where they were downloaded from NSX Manager and cached when NSX was registered as a solution. Depending on the number of hosts in the vSphere cluster, this process will take a few minutes to complete. Once the installation has completed, you will be presented with a screen like the one shown in the following screenshot:
The following figure depicts the management, control, and data plane components that make up an NSX implementation. Each has an important part to play in enabling ESXi for the Distributed Firewall and VXLAN. In this section, we will explore the interaction among the various components:
- vCenter server: This is the management component of the vSphere environment and is where the networking and security components of an NSX environment are all managed from.
- NSX Manager: This is the management plane of the NSX implementation. It integrates directly with vCenter and manages both the NSX controller cluster and the ESXi hosts. The NSX Manager is also responsible for pushing distributed firewall rules to each host that is prepared for the distributed firewall. In addition, the NSX Manager is also the API entry point for NSX operations via the REST protocol.
- ESXi Agency Manager (EAM): This is part of the vCenter deployment; it is responsible for installing the VIBs to each of the hosts.
When you prepare a vSphere cluster for NSX, the VIBs are copied directly from NSX Manager and cached by EAM. The EAM will then track the installation of each VIB onto each host in the vSphere cluster. If the VIB is not present, it is installed without the ESXi host requiring a reboot, and if it is present, a reboot is required to complete the upgrade.
Once the installation of VIBs has been completed, each ESXi host will have active TCP connections to the NSX Manager and NSX controller cluster. The connection to the NSX Manager is from the vsfwd daemon running on the ESXi host via the RabbitMQ message bus. The connection to the NSX Controller cluster is from the netcpa daemon running on the ESXi host via an SSL connection (TCP Port 1234). It is important that both channels of communication are active and can be checked via the communication channel health from each host, which is covered in a subsequent section:
When enabling a vSphere cluster for NSX in a brownfield environment, it is important to be cognizant that any preconfigured DFW firewall rules have the potential to impact virtual machines on the newly-configured vSphere cluster. Therefore, it is extremely important to ensure that the default Distributed Firewall rule remains as allow any any. Changing to deny before defining rules for allowing legitimate traffic from/to virtual machines will cause traffic blackholing.
As a best practice, vCenter server and virtual machines that require promiscuous mode should be excluded from the DFW if you are not planning to protect them. To learn how to exclude virtual machines from the DFW, refer to Chapter 6, Configuring VMware NSX Distributed Firewall (DFW) and SpoofGuard.
Installation of NSX VIBs that enable the Distributed Firewall and VXLAN are essential for a working NSX environment. This section will investigate how to manually verify that each VIB is installed correctly and whether communication to both the NSX controller cluster and NSX Manager are present.
The first control plane communication that we are concerned with is from the NSX Manager to each ESXi host via TCP port 5671. This port is reserved for the Rabbit MQ Message bus to the vsfwd daemon running on each host after the VMware Service Insertion Platform (VSIP) VIB installation, which is the Distributed Firewall kernel module. The NSX Manager uses the message bus to publish firewall rules down to each ESXi host. The ESXi host then applies them to vNICs of virtual machines that are running on top of its hypervisor.
The second control plane communication that is expected from each ESXi host is an open connection to each of the NSX controllers deployed. The NSX controller cluster is responsible for control plane information for ARP/MAC/VTEP tables. It is also used to program routes received on the Distributed Logical Router Control VM to each host (more on this in Chapter 2, Configuring VMware NSX Logical Switch Networks). From each host, we expect the netcpa daemon to have an active connection to the controller cluster on TCP port 1234.
To manually verify control-plane communication and VIB installation, you will need the following access to the following NSX components:
- SSH access to NSX Manager
- SSH access to each NSX controller
- SSH access to ESXi hosts that were prepared for NSX
You would not be expected to check communication of each and every host in your environment, as this can become unwieldly. However, this section is included for you to understand what the expected communication is, but in large deployments you would check the communication channel health per vSphere cluster as depicted in the earlier section.
To check whether the NSX VIBs have been installed successfully is crucial. The upcoming section details how to do this manually on an ESXi host and how to check NSX component communication.
In this section we perform manual verification that the VIBs have been successfully installed.
- SSH onto an ESXi host.
- Check whether VXLAN VIB modules have been installed by executing the following command:
esxcli software vib get --vibname esx-vxlan- You will receive an output similar to the following:
[root@vSphere:~] esxcli software vib get --vibname esx-vxlanVMware_bootbank_esx-vxlan_6.0.0-0.0.4987429 Name: esx-vxlan Version: 6.0.0-0.0.4987429Type: bootbankVendor: VMwareAcceptance Level: VMwareCertifiedSummary: Vxlan and host toolDescription: This package loads module and configures firewall for vxlan networking. ReferenceURLs:Creation Date: 2017-01-27Depends: esx-base >= 6.0, esx-base <= 6.5.0, nsx-api <= 2.1, vmkapi_2_3_0_0Conflicts: nsx-api = 2Replaces: esx-traceflow, esx-dvfilter-switch-security, esx-bfdProvides: com.vmware.vxlan = 1.0.0.0-nsx, com.vmware.switchsecurity = 1.0.0.0, com.vmware.traceflow = 1.0.0.0, com.vmware.bfd = 1.0.0.0Maintenance Mode Required: TrueHardware Platforms Required:Live Install Allowed: TrueLive Remove Allowed: TrueStateless Ready: TrueOverlay: FalseTags:Payloads: esx-vxlan
- If the module has been installed correctly, you should see open TCP connections on port
1234with the following command:
esxcli network ip connection list | grep 1234
An example is included below that shows the connection as established to each of the three NSX controllers from the point of view of an ESXi host:
- To see which NSX controllers the host is configured to communicate with, execute the following command:
cat /etc/vmware/netcpa/config-by-vsm.xmlThe following screenshot provides a truncated output of the command and its expected output.
- Check if
VSIP VIBmodules have been installed by executing the following command:esxcli software vib get --vibname esx-vsip:
- You will receive an output similar to the following:
[root@vSphere:~] esxcli software vib get --vibname esx-vsipVMware_bootbank_esx-vsip_6.0.0-0.0.4987429 Name: esx-vsipVersion: 6.0.0-0.0.4987429Type: bootbankVendor: VMwareAcceptance Level: VMwareCertifiedSummary: vsip moduleDescription: This package contains DFW and NetX data and control plane components.ReferenceURLs:Creation Date: 2017-01-27Depends: esx-base >= 6.0, esx-base <= 6.5.0, nsx-api <= 2.1, vmkapi_2_3_0_0Conflicts: nsx-api = 2Replaces: esx-vdpi Provides: vsip = 1.0.0-0Maintenance Mode Required: TrueHardware Platforms Required:Live Install Allowed: TrueLive Remove Allowed: TrueStateless Ready: TrueOverlay: FalseTags:Payloads: esx-vsip
- If the module has been installed correctly, you can open TCP connections on port
5671with the following command:
esxcli network ip connection list | grep 5671The following screenshot provides an example of the above command, the output shows a connection of established to the NSX Manager over TCP Port 5671:
- To see which NSX Manager the host is configured to communicate with, execute the following command:
esxcfg-advcfg -g /UserVars/RmqIpAddressThe following screenshot shows the results of the execution of the command above, and the expected configuration parameter is the IP address of the NSX Manager:
To check the communication channel between NSX Manager, NSX controller cluster, control plane agent, and Distributed Firewall agent, follow the following procedure:
- In the vCenter Web Client, navigate to
Networking & Security|Installation|Host Preparation - Select your vSphere cluster or an ESXi host
- Click on the
Actionsbutton and navigate to |Communication Channel Health:
The following screenshot displays the result of the preceding action and the ESXi communication health:
