In this chapter, we will see how to set up the various network resources that we will use in the next chapter. We will cover the following recipes:
Setting up an External Network
Creating 1,000 isolated networks without VXLANs
Making VXLANs work
Integrating Cisco 1000v into vCD
Giving your networks an Edge
Doing it all(most) without a Distributed Switch
Network virtualization is what makes vCloud Director such an awesome tool. However, before we go full out in the next chapter, we need to set up network virtualization, and it is what we will be focusing on here.
When we talk about isolated networks, we are talking about vCloud Director making use of different methods of the Network layer 3 encapsulation (OSI/ISO model). Basically, it's the same concept that was introduced with VLANs. VLANs split up the network communication in a network in different totally-isolated communication streams. vCloud makes use of these isolated networks to create networks in Organizations and vApps.
vCloud Director has three different network items listed as follows:
External Network: This is a network that exists outside vCloud, for example, a production network. It is basically a port group in vSphere that is used in vCloud to connect to the outside world. An External Network can be connected to multiple Organization Networks. External Networks are not virtualized and are based on existing port groups on vSwitch or a Distributed Switch (also called a vNetwork Distributed Switch or vNDS).
Organization Network: This is a network that exists only inside one organization. You can have multiple Organization Networks in an organization. Organizational networks come in three different types:
Isolated: An isolated Organization Network exists only in this organization and is not connected to an External Network; however, it can be connected to vApp Networks or VMs. This network type uses network virtualization and its own network settings.
Routed Network (Edge Gateway): An Organization Network connects to an existing Edge Device. An Edge Gateway allows defining firewall, NAT rules, DHCP services, Static Routes, as well as VPN connections and the load balance functionality. Routed Gateways connect External Networks to vApp Networks and/or VMs. This network uses virtualized networks and its own network settings.
Directly connected: This Organization Network is an extension of an External Network into the organization. They directly connect External Networks to vApp Networks or VMs. These networks do NOT use network virtualization and they make use of the network settings of an External Network.
vApp Network: This is a virtualized network that only exists inside a vApp. You can have multiple vApp Networks inside one vApp. A vApp Network can connect to VMs and to Organization Networks. It has its own network settings. When connecting a vApp Network to an Organization Network, you can create a router between the vApp and the Organization Network, which lets you define DHCP, firewall, NAT rules, and Static Routing.
To create isolated networks, vCloud Director uses Network Pools. Network Pools are a collection of VLANs, port groups, and VLANs that can use layer 2 in the layer 3 encapsulation. The content of these pools can be used by Organizations and vApp Networks for network virtualization.
There are four kinds of Network Pools that can be created:
Virtual eXtensible LANs (VXLAN): VXLAN networks are layer 2 networks that are encapsulated in layer 3 packets. VMware calls this Software Defined Networking (SDN). VXLANs are automatically created by vCloud Director (vCD); however, they don't work out of the box and require some extra configuration in vCloud Network and Security (refer to the Making VXLANs work recipe).
Network isolation-backed: These have basically the same concept as VXLANs; however, they work out of the box and use MAC-in-MAC encapsulation. The difference is that VXLANs can transcend routers whereas Network isolation-backed networks can't (refer to the Creating isolated networks without 1,000 VXLANs recipe).
vSphere port groups-backed: vCD uses pre-created port groups to build the vApp or Organization Networks. You need to pre-provision one port group for every vApp/Organization Network you would like to use.
VLAN-backed: vCD uses a pool of VLAN numbers to automatically provision port groups on demand; however, you still need to configure the VLAN trunking. You will need to reserve one VLAN for every vApp/Organization Network you would like to use.
VXLANs and Network isolation-backed networks solve the problems of pre-provisioning and reserving a multitude of VLANs, which makes them extremely important. However, using a port group or VLAN Network Pools can have additional benefits that we will explore later.
So let's get started!
Creating an External Network requires an existing port group in vSphere. This port group can be on a vSwitch, a Distributed vSwitch, or a Cisco 1000v Distributed Switch. The port group can be supported by a VLAN or a physical network.
Log in to vCloud Director with a system administrator (
Click on Manage & Monitor.
Click on External Networks.
Click on the green plus icon (+). Now, the New External Network wizard starts.
Select the vCenter that contains the port group and then select the port group you want the External Network connected to. If you have many networks, there is a filter just on the right above the list of the networks, as seen in the following screenshot:
Enter a name for this network and close the wizard.
An External Network is just a connection between vCloud Director and a port group on vSphere. vCloud Director adds IP management to the port group. When creating an External Network, you have to define a pool. This pool is used to automatically assign IP addresses to VMs, Edge Gateways, or vApp routers attached to this External Network. A Static IP Pool has to contain a minimum of one IP, but can contain the maximum available IPs minus the gateway address. vCloud Director will manage all the IPs assigned though Organization Networks and Edge devices. The IP assignments can be seen by right-clicking on the External Network and selecting IP Allocations as shown in the following screenshot:
Using only one IP in an External Network Static IP Pool is interesting only if all IPs for VMs are assigned manually and no Edge or vApp router is used. If this is not the case, one should assign at least 5 to 10 IPs to the Network Pool. We will make excessive use of the External Network and we will use its IP pool for load balancing, VPNs, and much more.
You can assign more than one IP range to an External Network, making it possible to create more than one IP range that can be used. However, IP allocation happens automatically and you are not able to control which IP from what range will be allocated to which specific VM. Creating multiple IP network ranges in External Networks is preferable when used together with IP suballocation in Edge devices.
When a VM is destroyed or undeployed, the IP will be released back to the pool. The setting of the default time for the IP release is set by navigating to Administration | General | IPaddress release timeout. The default value is
0 seconds. This setting specifies how long discarded IP addresses should be held before they can be reused. Think about your ARP tables and how long you have set your router's refresh time. If IP addresses are reallocated to new MAC addresses, a router might not be able to route it properly.
As I have already mentioned, we need one VLAN that is trunked to a Distributed Switch. The VLAN doesn't need to be routed. The only other requirement is that the network gear can accommodate a higher MTU.
In the system organization, we click on Manage & Monitor and then on Network pools.
Now click on the green plus (+) icon to add a Network Pool.
Now click on Network Isolation-backed as shown in the following screenshot:
Type in the VLAN number you would like to use for the Network Pool.
Now select the vCenter and the Distributed Switch you want to use.
Give the Network Pool a name.
After clicking on Next, you will see all the values you have entered.
Now we need to set the MTU for this VLAN to a minimum of
A safer choice is
1600, as this makes sure you have enough room for additional encapsulations down the track. Make sure that your physical switching infrastructure can use a higher MTU than the default
Click on the created Network Pool and select Properties.
Click on Network pool MTU and set the MTU to
1600as shown in the following screenshot:
Click on OK.
Network isolation-backed networks actually don't use layer 2 and layer 3 encapsulations, but they use the MAC-in-MAC encapsulation. When a new vApp or Organization Network is created, vCD will create a new port group and will then use this port group to encapsulate the traffic on a MAC basis. The same technique was used in VMware Lab Manager, which was then called Host Spanning Networks. This doesn't come without cost. Because of the additional encapsulation, another 24 bits are required for each package, meaning that the MTU should be increased to a minimum of
1524. If you don't change the MTU, you will have a network frame fragmentation.
The good thing is that Network isolation-backed Network Pools are quite fast and easy to configure and set up. They provide you with 1,000 isolated networks for each VLAN. You can define more than one Network isolation-backed network. However, you can only assign one Network Pool to an Organizational virtual Datacenter (OvDC), as there is a one-to-one relationship between them. You cannot create isolated networks before you assign a Network Pool to an OvDC.
Navigate to Manage & Monitor | Organizational VDC.
Right-click on the OvDC you want the pool assigned to and select Properties.
Click on Network pool & Services.
Select the network pool you like to assign as shown in the following screenshot:
Select the number of networks you would like to assign to the OvDC as shown in the following screenshot.
One of the disadvantages is that the networks are isolated, meaning we can't use them for anything other than vCloud Director.
We will work extensively with vApp and Organization Networks in Chapter 2, vCloud Networks
VXLANs are great, but they don't work out of the box. In the following sections, we discuss how to set them up.
As you already have vCloud set up, you must have a vCloud Network and Security appliance (vCNS) deployed (formally known as vShield), and it should be configured to use your vCenter. For this recipe, you will need to be able to log in to the vCNS appliance with an administrator account.
Additionally, we need a VLAN on which the VXLANs will exist, and having a DHCP in that VLAN makes things easier. If no DHCP is accessible on this VLAN, you will need to provide one IP address per ESXi server in this VLAN.
The Segment ID you have to enter in step 14 in the How to do it… section is rather important, especially when you have multiple vCNS or vCloud installations (not multiple cells). Each of these installations should have a different range. If this is your first VXLAN installation, just use the range that is supplied in the steps.
Last but not least, you should have a multicast address range (see http://en.wikipedia.org/wiki/Multicast_address); this is best arranged with the network administrator. If you can't figure out what to use, the range given in the steps will work fine for a VXLAN that exists only in one location.
Open a browser and browse to the vCNS appliance
https://[ip of vCNS].
Log in to the appliance (the default username is
adminand the password is
Make sure that you have switched to the Host & Clusters view.
Click on your data center.
Now select Preparation and then Connectivity as shown in the following screenshot:
Click on Edit.
Select your Cluster.
Select the Distributed Switch as well as the VLAN ID for the VXLAN that you want to use and click on Next as shown in the following screenshot:
Now select a Teaming policy (for example, Fail Over) and its MTU (for example, 1600) as shown in the following screenshot:
Click on Finish.
Click on Segment ID and then on Edit as shown in the following screenshot:
You have to now enter a range for the Segment IDs (for example,
Now enter the Multicast address range (for example,
Click on Finish.
We are now done with vCNS and can leave the rest to vCloud Director.
Log in to vCloud Director and click on Network pools.
VXLANs were created by VMware together with Cisco. The idea was to solve the problems of modern data centers. Typically, these problems relate to the inflexibility of VLAN and Switching boundaries due to too much subnetting, IP, and VLAN management. The idea behind VXLANs is to create virtualized networking that is used on top of the common networking layer. They are in use just like the Network isolation-backed Network Pools we discussed in the other recipe; however, VXLANs have the benefit of being routable, flexible, and can transcend to different locations. This makes them extremely flexible and elegant to use.
VXLANs are actually like VLANs; the main difference is that VLANs (802.1q) have a 12-bit namespace whereas VXLANs have a 24-bit one, which increases the number of VLANs from 4,096 to more than 16 million unique namespaces.
VXLANs use layer 2 in layer 3 encapsulation. This means they use the Internet Protocol (IP, layer 3) to propagate the networks (from layer 2 upward), making them routable and far more flexible across network borders. One could envision VXLANs as a tunnel between two endpoints where additional networks exist.
VXLANs don't really exist in vCloud Director; VXLANs are defined in the vCNS appliance. vCNS creates the VXLAN tunnel endpoints and manages the VXLANs for the whole virtual infrastructure. For all this to happen, vCNS must install an agent on each ESXi server. This is done when you click on Finish, as explained in step 12 in the How to do it… section of this recipe. These agents provide the connection between VMs and the VXLANs. As the VXLAN packages are bigger than the common network packages, we have to adjust the MTU to avoid frame fragmentation. A safe setting is 1600. Each agent will be deployed and connected to a new vmknic. The IP for the vmknic is assigned via DHCP; however, this can be changed in vCNS for each ESXi server.
The Fail Over policy that you set on the Distributed Switches depends on what the physical switching architecture can do. If EtherChannels are set up, choose them. Link Aggregation Control Protocol (LACP) can be chosen either in active or passive mode. LACP in an active mode sends out packages to talk to LACP-activated devices, whereas in a passive mode it waits until a LACP-activated device talks to it. If in doubt about all of this, please involve your network team.
If you want to configure VXLANs with Static IPs for each vmknic, follow the ensuing procedure:
Navigate to Hosts & Clusters View | Datacenter | [your datacentre] | Network Virtualization | Connectivity as shown in the following screenshot:
Write down which vmknic is used for VXLANs.
Log in to vCenter as an administrator.
Set a fixed IP for each of the vmknics on each ESXi server.
The following are some links for further reading:
VMware's easy to read explanation at http://www.vmware.com/solutions/datacenter/vxlan.html
Cisco's Digging deeper into VXLAN at http://blogs.cisco.com/datacenter/digging-deeper-into-vxlan
A good case study on VXLANs at http://it20.info/2012/05/typical-vxlan-use-case/
The Cisco 1000v Distributed Switch is an alternative to the VMware Distributed Switch; however, getting it working with vCloud is a challenge. In the following sections, we will see how to overcome it.
First and foremost, we need the Cisco 1000v installed (refer to the How it works section in this recipe). The next important thing is to hook the Cisco 1000v up to vCenter (refer to the See also section links in this recipe). After all this is done, you can use the Cisco 1000v in vSphere and we can now start linking it up to vCloud.
Log on to the Cisco 1000v Virtual Supervisor Module (VSM).
Run the following commands to activate the REST interface of the VSM:
1kv# conf t 1kv (Config)# feature network-segmentation-manager 1kv (Config)# feature segmentation 1kv (Config)# exit 1kv # copy running-config startup-config [########################################] 100% 1kv# exit
Log out of the Cisco 1000v.
Log in to the vCNS (vShield) as an administrator.
Click on Settings & Reports.
Click on Networking and then on Add Switch Provider as shown in the following screenshot:
Enter the admin credentials for the VSM and click on OK.
The Cisco 1000v replaced the VMware Distributed Switch. The benefit of this is that you can use the Cisco tools and Cisco language to configure it. For all intents and purposes, it acts and behaves like a Cisco physical switch. One of the drawbacks is that there can be maximal 63 hosts (63 hosts and one Cisco v1000) connected to one Cisco 1000v and the integration between Cisco and VMware isn't as smooth as it may appear to be.
Cisco 1000v consists of two different items. The VSM is the management console and is basically a VM where the Cisco OS runs. The Virtual Ethernet module (VEM) is a plugin into the ESXi server that replaces the functionality of the VMware Distributed Switch.
There are basically two different designs for the Cisco 1000v installation. The first and easiest is to install the Cisco 1000v VSMs (Manager) as a VM on vSphere. This has considerable drawbacks as the Cisco 1000v VSMs can't be used with the Cisco 1000v itself, meaning you need to install it using normal vSwitches, which in return calls for a separate management cluster. The other better but more costly alternative is to use Cisco 1010. The Cisco 1010 is nothing else but a physical server that can contain up to six VSMs.
vCloud Director talks to vCenter for management (create VMs and so on), and talks to vCNS to create and manage complex networks (for example, Edge). The vCNS communicates with vCenter to create port groups and push out configurations to Distributed Switches. vCenter communicates with ESXi servers to facilitate management and to push network configurations out to the ESXi servers.
When the Cisco 1000v enters the picture, things change. Instead of vCNS talking to vCenter (the dotted line in the preceding figure), it now talks to the Cisco 1000v VSM. The VSM will then talk to vCenter and the VEM installed in the ESXi server to facilitate networking.
The following are some useful links for the Cisco 1000v:
Technical documentation at http://www.cisco.com/en/US/products/ps9902/tsd_products_supPort_series_home.html
Deployment Guide for the 1000v at http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps9902/guide_c07-556626.html
Implementing VXLAN with vCloud Director (more details) at http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps9902/deployment_guide_c07-703595.html
For the recipes in the next chapter, we need an Edge. So, let's see how it works.
One External Network, with some free IP addresses
One OvDC connected to one Network Pool with at least one free isolated network
Log in to vCloud (not the organization) as a
Double-click on the organization you want to create the Edge in.
The Organization Network should now have opened as a separate tab.
Click on Administration.
Double-click on the OvDC your Network Pool is associated with.
Click on Edge Gateways as shown in the following screenshot:
To create a new Edge device, click on the green plus (+) sign.
The Edge wizard opens up and the first page lets you choose some very basic and important settings. I marked the settings that I will use for this installation so that it matches the need of the Edge device in the next chapter. All of them can be configured later too.
Perform the following steps for configuring the Edge Gateway:
Of the two Edge gateway configuration options, Compact and Full, select Compact. This option basically decides the resources that should be allocated to the gateway. Compact can be later upgraded to Full.
Uncheck the Enable High Availability option; if chosen, the gateway is protected against faults in ESXi hosts.
Check the Configure IP Settings option; it is used to manually configure the IP setting for the Edge.
Check the Sub-Allocate IP Pools option; it makes the IPs from the External Network available to the Edge.
Uncheck Configure Rate Limits; it is used to reduce the inbound and outbound bandwidth, as shown in the following screenshot:
Perform the following steps for configuring the External Networks:
Perform the following for configuring the IP settings:
For each External Network, select if the gateway IP should be automatically taken from the pool or assigned manually.
Clicking on Change IP Assignment will open up a window where you can assign the IP manually.
Select the External Network you would like to create a suballocation for.
Type the range you want to suballocate.
Click on Add.
Name the Edge device and click on Next to see the summary, or Finish to start the deploy of the Edge device.
We now need an organization network connecting to the Edge. Click on Org VDC Networks.
Click on the green plus (+) sign to create a new Organization Network.
Select Create a routed network…, select the Edge device you have created, and click on Next.
Type in the addresses in the Gateway address and Network mask options.
The Edge can forward your DNS requests. If you don't want that and have your own DNS server in this Organization Network later configured, you can switch it off.
Specify a Static IP pool for this network.
Give the network a name and description.
Click on Next for a summary or Finish to create the Organization Network.as shown in the following screenshot:
The Edge is essentially a router with extras. It provides you with the ability to route between different networks, create firewall rules, create a DHCP service for Organization Networks, create load balancers, and also define a VPN network here.
The Edge devices are only accessible from the organization they were created in. The Edge has two sides: the northbound facing (External Networks) and the southbound facing (Organization Networks). For each Organization Network, the Edge provides a gateway address that can also act as a DNS forwarder. All communication through the Edge has to pass the firewall and can be controlled that way.
Each Organization Network can be configured with D-NAT and S-NAT, as well as DHCP and Static Routing. Load balancing can be configured and we will do that in the next chapter. As the Edge can be connected to multiple External Networks, it can serve as a hub for all kinds of connections, for example, direct Internet connections and local networks.
I would strongly encourage you to take a trip into the vCNS and look at the Edge we created, and compare its vCNS options to the options vCloud Director presents you.
Log in to vCNS.
Expand Datacenter and click on your data center.
Select Network Virtualization and then Edges.
Double-click on your Edge device as shown in the following screenshot:
We will work with the Edge in the next chapter and find out what we can do with it
So you want vCloud, but you don't have a Distributed Switch; here is how you can enjoy vCloud using vSwitches.
We discuss the changes required for each of the different networks:
Network Pools: The only Network Pool type you can use is the port group-backed one. This also means that you will need to have one port group per isolated network. The drawback is that you can also only assign one Network Pool to each OvDC, making it a one-to-one relationship between the port group, OvDC, and isolated network.
Using vSwitches instead of Distributed Switches with vCloud Director is a possibility, but normally this method is only used when VMware Enterprise Plus Licensing is not available. However, with the introduction of vCloud Suite (which comes with vSphere Enterprise Plus), it is not really needed anymore. Still, in some cases this might provide a solution.